(200) Scalable Natural Language Processing for Large-Scale Covariate Adjustment in Healthcare Database Studies

Background: Unstructured clinical notes in electronic health records (EHR) have potential to supplement claims data for covariate adjustment but natural language processing (NLP) is required to leverage this information.

Objectives: To evaluate predictive performance characteristics and covariate balance when comparing different types of NLP-derived EHR features to supplement Medicare claims when modeling the propensity score (PS) and 1-year risk of GI bleed for patients initiating high- vs. low-dose proton-pump inhibitors.

Methods: We applied three NLP methods (n-grams, MTERMS-a linguistics-focused approach, and Bidirectional Encoder Representations from Transformers (BERT) Sentence Embeddings) to generate large numbers of structured text features from the free-text clinical notes. We considered 6 covariate sets when fitting PS models: 1) Prespecified Variables; 2) Set 1 + Claims Codes; 3) Set 2 + EHR Codes; 4) Set 3 + n-grams; 5) Set 3 + MTERMS; 6) Set 3 + Sentence Embeddings. We evaluated the cross-validated predictive performance for treatment assignment and standardized covariate differences across treatment groups for all covariates, regardless of whether they were included in the adjustment set. We also evaluated predictive performance when modeling outcome risk.

Results: When modeling treatment assignment, there is substantial improvement in the predictive performance when adding thousands of additional codes from both claims data and EHRs (C-statistic of 0.72 for Covariate Set 3 vs 0.65 for Covariate Set 1). When including thousands of additional NLP generated features using Ngrams, in addition to prespecified variables and structured codes, there were additional improvements when predicting treatment assignment (C-statistic of 0.78 for Covariate Set 4). Results further show that when prespecified variables are adjusted for in isolation, there are still substantial imbalances in many features from claims codes, EHR codes, and NLP-generated features. As more features are added, overall balance improves, with Covariate Set 4 providing the best overall balance, with standardized differences for all features (across all covariate sets) close to 0.1. However, when modeling outcome, there was no meaningful improvement in terms of prediction when supplementing pre-specified variables with thousands of additional codes or NLP generated features.

Conclusions: Results indicate that controlling for large numbers of automatically selected variables can improve covariate balance in healthcare database studies beyond adjusting for investigator specified variables. The extent to which this improves confounding control will be explored in future work.