(330) A Comparison of Natural Language Processing (NLP) vs Clinician Review for Identification of Myocarditis Cases Following Vaccination with mRNA-1273

Background: Myocarditis is a rare event that recently received heightened media attention after cases were reported following administration of mRNA COVID-19 vaccinations. Post-marketing surveillance data capture these reports and include a narrative field containing clinical notes. Clinicians review the critical details in the narrative to confirm myocarditis cases using pre-specified case definitions.

Objectives: To evaluate the concordance between myocarditis cases identified through NLP with cases confirmed by clinical review.

Methods: Rule-based algorithms were developed by a team of pharmacovigilance clinicians, epidemiologists, and data scientists following the CDC case definition for acute myocarditis (probable, confirmed), by translating symptoms (such as chest pain, dyspnea, palpitations, syncope, myocardial edema, and painful respiration), and laboratory and diagnostic test results (elevated troponin, abnormal electrocardiogram, echocardiogram, and cardiac magnetic resonance imaging (cMRI), among others) into terms according to the Medical Dictionary for Regulatory Activities (MedDRA v.24). Algorithms were applied to the narratives of reported cases of myocarditis in the post-marketing safety database from 2021-2022 that were clinically reviewed and categorized according to the CDC myocarditis criteria.

Alternate terms and spellings were identified through an iterative process. Multi-channel keyword capture was performed consisting of an exact match, fuzzy match, pattern match, and unit conversion for lab values. Negation and temporal terms were also flagged.

Results: Using clinicians’ reviews of 652 narratives from 2021 as a gold standard, the NLP rule-based algorithm achieved 77% accuracy, and 84% sensitivity. Compared to clinicians’ review of 485 narratives from 2022, accuracy was slightly lower at 72% but sensitivity increased to 86%.

Conclusions: Clinician review of free-text fields is the gold standard for confirming myocarditis cases and assigning CDC classification levels. Natural language processing methods can help to prioritize case review by flagging entries that are likely to be true positive cases. Further refinement of algorithms and supplementation with machine learning techniques can enhance the sensitivity and specificity.