Using External Validity Methods to Improve the Precision of Effect Estimates in Subgroups

Background: Subgroup analyses are common in both randomized controlled trials (RCTs) and observational research; unfortunately, restriction to subgroup members to test for heterogeneity can yield false positives and imprecise effect estimates. Additionally, testing for differences with restriction is optimal for bias and precision only if A) the true effect is equal in members and non-members or B) the true effect differs between members and non-members due to subgroup membership itself (or unmeasured effect measure modifiers [EMMs]). If, on the other hand, the true effect differs between members and non-members due to different distributions of measured EMMs (not unmeasured EMMs or subgroup membership itself), combining information from members and non-members with external validity methods can improve precision without bias.

Objectives: Demonstrate how external validity methods (e.g., inverse odds weights) can improve precision of subgroup treatment effect estimates in real RCT data.

Methods: We obtained data from the PRIME RCT of panitumumab in patients with metastatic colon and rectal cancer from Project Datasphere™. We focused on the Hispanic patient subgroup due to a differing distribution of KRAS (a gene related to treatment efficacy). We weighted non-Hispanic White patients to resemble Hispanic patients in 6 measured potential EMMs (e.g., age, KRAS distribution, sex), combined Hispanic and weighted non-Hispanic White patients in one data set, and estimated the effect of treatment on 1-year progression-free survival (PFS) in the combined cohort. We obtained percentile-based 95% confidence limits from 1,000 bootstraps. To demonstrate potential bias when subgroup membership modifies the effect of treatment after accounting for other EMMs, we also reweighted mutant-type KRAS (no treatment benefit) patients to resemble wild-type KRAS (treatment benefit) patients.

Results: The PRIME RCT included 795 non-Hispanic White and 42 Hispanic patients with complete EMM data. While the Hispanic-only analysis estimated a 17% (95% C.I. 8.8%, 45%) increase in one-year PFS with panitumumab, the combined weighted estimate was more precise (8.7%, 95% CI -5.3%, 22%) while differing from the full population estimate (-1.0%, 95% CI: -7.5%, 5.9%). When targeting wild-type KRAS patients the combined weighted estimate incorrectly suggested no benefit.

Conclusions: External validity methods using information from non-members of subgroups generate more precise subgroup estimates without any bias if treatment effect variation between subgroup members and non-members arises from measured EMMs. However, if subgroup membership itself is responsible for treatment effect variation or unmeasured EMMs differ in distribution between members and non-members bias can occur.