Establishing a validation framework of claims-based treatment discontinuation definitions using natural language processing and electronic health records

Background: The estimation of treatment discontinuation in claims database research primarily relies on the gaps between prescription refills, but such definitions are rarely validated.

Objectives: To establish a framework incorporating natural language processing (NLP) aided chart review in validation of claims-based treatment discontinuation of antipsychotic medications (APMs), benzodiazepines (BZDs), warfarin, and direct oral anticoagulants (DOACs) using electronic health records (EHRs) linked with claims data.

Methods: We linked EHRs from the Mass General Brigham with Medicare Part D data from 2007 to 2020. A rule-based NLP-aided chart review algorithm was applied to determine the status of taking a medication as the reference standard, which was then validated against manual chart review. The date when medication-taking status was determined was the index date. We extracted the prescription records of medications in the 180 days before and after the index date from the claims data to estimate the dispensing gap, measured as the days between the end of last pre-index prescription and the start of first post-index prescription. A dispensing gap larger than 15, 30, 45, 60, and 90 days, respectively, was used as the claims-based definitions of treatment discontinuation. We calculated the sensitivity, specificity, positive predictive values (PPVs) and negative predictive values (NPVs) of claims-based definitions, when compared to the reference standard.

Results: Our study cohort included a total of 573 haloperidol, 15,504 atypical APM, 38,845 BZD, 23,529 warfarin, and 4,004 DOAC prescription records. Sensitivity of the claims-based definitions of treatment discontinuation was generally modest ( < 0.6) across the study medications. Specificity ranged from 0.54 (15-day gap) to 0.79 (90-day gap), 0.62 to 0.85, 0.45 to 0.75, 0.64 to 0.87, and 0.48 to 0.79 for haloperidol, atypical APMs, BZDs, warfarin, and DOACs, respectively. Informed by the published discontinuation rates of haloperidol (80%), atypical APMs (65%), BZDs (90%), warfarin (70%), and DOACs (60%), the PPVs ranged from 0.87 (15-day) to 0.92 (90-day), 0.76-0.87, 0.93-0.95, 0.75-0.83, 0.73-0.86, respectively; NPVs, not varying by the grace periods, were 0.33, 0.47, 0.16, 0.53, and 0.37, respectively.

Conclusions: The optimal gap-based treatment discontinuation definitions based on claims data may differ by medications. We have established an efficient validation framework using NLP-aided chart validation and demonstrated the scalability in several medication classes.