A Combined Machine Learning and Natural Language Processing Approach to Determine Metastatic Date Through Integrated Temporal Modeling

Background: Natural Language Processing (NLP) is a promising technique to extract information from
unstructured EHR documents and enrich real world data (RWD) in oncology. Accurately extracting event
dates with NLP remains an open research area due to the variety of ways dates can be recorded, linking
of specific date entry to the event in question in a document, and the complex patient visitation timeline
where an event is referred to many times for a variety of reasons.

Objectives: The objective of this study is to use Machine Learning (ML) to improve accuracy of NLP-
extracted metastatic date of diagnosis (mets date) in EHR-based RWD.

Methods: ConcertAI RWD was used to train ML and NLP models for mets date in Non-Small Cell Lung
Cancer (NSCLC). Labels were generated for curated mets date from unstructured EHR notes. To train the
model, possible mets dates, along with contextual information from patient documents, were extracted
using NLP. Model features were developed from density of lab testing and cancer medication dates
before and after rank ordered NLP extracted mets dates. An XGBoost classifier was trained on these
features to select the optimal date called the integrated ML (IML) mets date. Patient360TM dataset
comprising of expert curated clinical variables served as the ground truth (GT) for validation. Real-world
overall survival (rwOS) estimated from mets date until date of death, was compared between GT versus,
NLP-extracted mets date and the IML estimated mets date. Model performance was also assessed by
subgroups of age, gender, ECOG, and histology to assess bias induced by covariates.

Results: 11,192 metastatic NSCLC patients from Patient360TM were included. Median rwOS from the GT
mets date was 16.9 mos (95% CI: 16.4-17.5) compared to 13.0 mos (95% CI: 12.5-13.6) with NLP
extracted mets date (diff=4.4 mos, logrank p<.005) and 17.5 mos (95% CI: 16.9-18.1) with IML estimated
mets date (diff=0.6 mos, logrank p=0.76). The adjusted standardized mean difference (aSMD) were
0.1062 for GT vs. NLP and 0.0014 for GT vs. IML. Stratified analysis did not identify any differential
results by subgroups. aSMDs for subgroups comparing GT vs. IML were between 0.001 to 0.09 and
median rwOS differences with p>0.05.

Conclusions: ML-based models trained on expert curated RWD can significantly enhance the accuracy of
NLP extracted date by leveraging features from structured EHR when generating RWD from
unstructured EHR documents. This approach can be scaled across multiple date events to capture the
temporal sequence of health events and generate meaningful insights about cancer patients’ journey.
Future applications of this work across tumor types can provide a reliable approach to scale rapid
insights from EHR documents.