Adjusting number of cases, risk, risk difference and risk ratio estimated through an algorithm with unknown sensitivity: a contribution from the ConcePTION project

Background: In database studies, outcomes of interest are retrieved using algorithms that may be affected by errors. It is recommended to choose a specific algorithm, which, however, may underestimate the number of cases and risk. In association studies, the absolute value of risk difference may be underestimated, and the validity of risk ratio relies on the assumption that sensitivity (SE) is consistent across exposure groups. Estimating SE is typically unfeasible.

Objectives: To provide and test formulas to adjust study estimates based on positive predictive values (PPV).

Methods: Performing a validation study on SE and specificity (SP) is rarely feasible. On the other hand, validation studies may enable the estimation of PPV. Consider the main algorithm (A) for a given outcome; a screening algorithm for A is a secondary algorithm (B) for the same outcome with lower SP but higher SE. If PPVs and prevalence of both A (PPV_A and P_A) and B (PPV_B and P_B) are available, SE of A (SE_A) can be estimated by SE_A <= (P_A x PPV_A)/(P_A x PPV_A + P_B x PPV_B), and, relying on some assumption, the formula becomes an equality. Based on this, we derived formulas to adjust number of cases and risk. In association studies, if PPVs are available across exposure strata, we derived formulas to adjust risk difference under multiple scenarios, and to adjust risk ratio without assuming non-differentiality.
For example, if N_A and N_B are the number of cases detected by A and B, respectively, the true number of cases can be calculated as follows N >= N_A x (PPV_A)/(1- (N_B x PPV_B / (N_A x PPV_A + N_B x PPV_B))), and, under some assumptions, this becomes an equality. The proposed methodology is applied to a published validation study, where cases of angioedema seen in an emergency room were retrieved from administrative records using ICD9CM 9951 (algorithm A) or ICD9CM 9950, 99527, 7088, 7080, 7081, 7089, 37633, 37482, 47825, 4786, 47875, 5088, 7823, 9952 (algorithm B).

Results: Cases retrieved from the data were N_A = 34 and N_B = 451. After validating all cases of A and a sample of cases of B, PPV_A was 35.3%, and PPV_B was 8.3%. Using our formulas we obtained SE_A <= 24% and N >= 49.

Conclusions: Whenever a complementary screening algorithm can be defined, the estimates of PPV of both algorithms provide tools to reduce or remove underestimation of cases and rates, and to reduce or remove bias in risk difference and risk ratio. This new methodology has the potential to improve evidence based on data obtained through the reuse of real world data.