(055) Development and portability of a text mining algorithm for capturing disease progression in electronic health records of with stage IV non-small cell lung cancer treated with immuno-chemotherapy

Background: Manual chart review of unstructured text fields within electronic health records (EHR) for disease progression is time-consuming, making it unsuitable for large prospective multicenter cohort studies. Text mining could be an efficient alternative for capturing progression events from EHRs.

Objectives: To develop and evaluate the portability of a text mining algorithm for prospectively capturing disease progression in EHR data of metastatic non-small cell lung cancer (mNSCLC) patients treated with immuno-chemotherapy.

Methods: This multicenter study used EHR data from mNSCLC patients receiving immuno-chemotherapy between 01/10/2018 and 31/12/2022 in four Dutch hospitals. A text mining algorithm for disease progression identification was developed in hospital 1 and hospital 2, and then transferred to hospitals 3 and 4 to evaluate its portability. Performance metrics (F1-score, recall/sensitivity, precision/positive predictive value, specificity, and negative predictive value) were calculated by comparing the outcomes of the text mining algorithm to manual chart review. Also, data were simulated to come available over time to assess performance in real-time application. Median progression free survival (mPFS) was calculated for all four hospitals using the Kaplan-Meier method both after text mining and manual chart review.

Results: During both development and portability, the text mining algorithm performed well in identifying disease progression, with all performance scores >90%. When real-time performance was simulated, the performance scores in all four hospitals exceeded 90% from 15 weeks after start follow-up. Although the exact progression dates identified with text mining and manual chart review varied in approximately 30% of all patients with disease progression, the number of patients labeled with progression too early (n = 24) and too late (n = 22) was well balanced with discrepancies ranging from -116 to 384 days. Nevertheless, for each hospital, the PFS curves constructed with text mining and manual chart review were highly similar.

Conclusions: In this study, an accurate text mining algorithm for capturing disease progression in EHR data of patients with mNSCLC was developed. The algorithm was portable across different hospitals and the performance over time was good, making this an interesting approach for real-time multicenter cohort follow-up.