(211) Machine Learning Natural Language Processing Classifiers to Enhance Opioid Overdose Surveillance Precision

Background: In the last twenty years, the United States has witnessed a fivefold increase of drug overdose fatalities, including a 14% escalation in the latest reporting year. Kentucky, notably ranking fourth in opioid-related deaths, exemplifies the national distress. The surge particularly impacts historically marginalized communities. While Emergency Medical Services (EMS) data can provide a pivotal role in overdose public health surveillance, the validity of existing case definitions remains unverified.

Objectives: This investigation assesses the accuracy and demographic parity of five opioid overdose definitions within EMS data and explores the efficacy of machine learning natural language processing (MLNLP) classifiers as innovative alternatives to enhance surveillance precision.

Methods: In a thorough analysis of Kentucky EMS records from 2018 to 2022, our study undertook a classification difficulty-weighted, randomized review of 2,483 emergency encounters, ensuring demographic representation with half of the cases involving Black patients. Expert paramedics annotated encounter narratives with ground-truth labels. A tuned Random Forest MLNLP model was employed to evaluate term frequency-inverse document frequency (TF-IDF) transformed narratives, benchmarking various definitions against key performance metrics, with a focus on the F-score (the harmonic mean of precision and recall) for accuracy and equity assessment.

Results: The investigation revealed significant disparities in the stringency of definitions, with the strictest yielding a 41% sensitivity, and the most inclusive at 94%, albeit with a 54% positive predictive value (PPV). The MLNLP model demonstrated superior performance, with an F-score of 0.891, markedly outperforming existing definitions. Across definitions, racial disparities were evident, with consistently lower F-scores for Black patients. Applying the MLNLP model to the full EMS dataset revealed that it uniquely identified over 25,000 cases not detected by current definitions, which constitutes one-fifth of all cases, underscoring the transformative potential of MLNLP in public health informatics.

Conclusions: The study underscores the limitations of conventional rule-based models and showcases the advanced capabilities of MLNLP in opioid overdose surveillance. It offers empirical evidence for jurisdictions to refine surveillance strategies and emphasizes the critical need for demographic inclusivity in ML models to ensure equitable public health interventions. The unique identification of a substantial number of cases by MLNLP also signifies its value in capturing a broader scope of the opioid crisis.