Using QBA in the Selection of Computable Operational Definitions to Minimize Misclassification & Underpower in RWE Studies

Background: Stakeholders of real-world evidence (RWE), including regulatory agencies, have emphasized the need for clear operational definitions for each study element. However, clinical concepts can be operationally defined in many ways, even after application of common data models. There is currently no unified validation metric that is widely accepted, and small differences in computable operational definitions (CODefs) can have a large impact on misclassification, which needs to be accounted for in the study design. Therefore, thoughtful selection of CODefs is critical to ensure internal validity and assess the broader use across data sources and studies in global RWE programs.

Objectives: To demonstrate the use of validation metrics in guiding the selection of CODefs for a study outcome variable and evaluate impact of the CODef selection on study power using Quantitative Bias Analysis (QBA).

Methods: We performed a literature review to identify CODefs for heart failure (HF) and hematological malignancy (HM). Published validation metrics were identified via literature search and extracted to provide context for each CODef. These validation metrics were used as inputs in QBA-based power calculation to determine the initial target sample size for a cohort study where the outcome variable was defined using each of the identified CODefs. The following assumptions were made for the sample size calculations: 90% power, odds ratio (OR) of 0.3 as the minimum detectable effect, type I error rate of 5%, and an event rate amongst the untreated of 0.004 per person-month.

Results: A total of 9 CODefs for HF and 2 for HM were identified in the literature review. These CODefs were composed of diagnosis records, medication records and lab results (6 code lists including 1329 total codes), in various clinically-relevant combinations. The CODefs were generally associated with high specificity for both HF & HM (range: 98.9 - 99.9%); although one outlier HM CODef with lower specificity was seen (range: 73 - 99.8%). More variation was seen in sensitivity (range for HF: 41.6 - 78.7%, range for HM: 52.2 - 99.0%). The QBA-corrected sample size estimates showed the choice of CODef for outcome derivation could impact study power by 11.1 - 75.0% for HF and 36.1 - 83.3% for HM. A reduction in specificity of a CODef had a greater impact on study power then a similar reduction in sensitivity.

Conclusions: The selection of an appropriate CODef will need to be guided by its intended use within a study, and the need to reduce bias and improve study power. We demonstrate the impact of the choice CODef for an outcome variable on study power using QBA. It is critical to consider validation metrics in selecting CODefs for RWD studies; therefore, we recommend this QBA approach when selecting a CODef for key study elements.