Evaluating concordance in results from database studies and randomized trials after aligning distribution of key risk factors for the outcome of interest

Background: The RCT-DUPLICATE initiative emulated over 30 randomized clinical trials (RCT) using claims databases. While concordance in results was high when RCT design could be emulated closely, design emulation differences and bias contributed to divergence in results for several RCT-database study pairs. Differences in the distribution of key risk factors for the outcome of interest between the RCT and database study have been suggested as contributors to divergence in effect estimates.

Objectives: To evaluate concordance in treatment effect estimates between three RCTs and database studies before and after aligning the distribution of key risk factors for the outcome of interest.

Methods: Using 3 large US claims databases (2012-17), we emulated 3 RCTs: TECOS, SAVOR-TIMI 54, and CARMELINA. Each evaluated the effect of dipeptidyl peptidase-4 inhibitors (DPP-4i) versus placebo on risk of major adverse cardiovascular outcomes (MACE). We used second generation sulfonylureas (SU) as an active-comparator proxy for placebo, since the SU glimepiride had no meaningful effect on MACE in the CAROLINA trial, compared to the DPP-4i linagliptin. We adjusted for >100 measured confounders using 1:1 propensity score (PS) matching. We evaluated hazard ratios (HR) and rate differences (RD) in the 1:1 matched cohorts before and after using raking weights to align the database study population to the respective trial populations. We evaluated raking weights based on age and sex only, as well as including baseline atherosclerotic cardiovascular disease (CVD), heart failure, metformin and/or insulin.

Results: Of the 3 trials, the characteristics of the SAVOR-TIMI 54 trial were the most different from the database study in terms of baseline CVD (myocardial infarction 38% vs 1%; atherosclerotic CVD 78% vs 20%) and demographic distributions. The RCT HR was [1.00 (0.89, 1.12)]. After PS matching, the HR was 0.78 (95% CI: 0.73, 0.83); using raking weights to align the age/sex distributions resulted in a HR of 0.75 (0.70, 0.80), additionally aligning other risk factors resulted in HR of 0.92 (0.60, 1.43), closest to the trial estimate. However, the weight distribution was wide, with a mean of 1 and max 297. RDs showed similar patterns. For TECOS and CARMELINA, weighting did not meaningfully change the HRs and RDs.

Conclusions: Different population distributions of key risk factors could explain some divergence in results between RCTs and database studies. Raking weights can align the marginal distributions of compared populations, but bias variance tradeoffs need to be considered as large weights can increase variance.