(203) Testing machine learning prediction models to impute missing data in claims databases

Background: Missing data are common in claims data. Important behavioral confounders such as smoking are rarely captured due to the absence of financial incentive to provide a diagnosis code. Smoking status – categorical – is recorded “yes” for a fraction of smokers and otherwise left missing. Thus, only one category of the variable is observed (“yes”) and standard multiple imputation models cannot be fitted. A common way to handle missing values is to treat missing values as “no”, which may lead to substantial misclassification. Despite the wide use of claims data and the prevalence of missing data, few studies explored methods to impute data of this missing pattern. Using an external source of information to enrich claims database could allow for imputation using machine learning techniques.

Objectives: To explore the feasibility of imputing missing values of behavioural confounders using a machine learning approach and taking the example of smoking status.

Methods: A simulation study was conducted using primary collected cohort data (NOVELTY cohort including 12,224 people with asthma and/or chronic obstructive pulmonary disease – NCT02760329). In this cohort, the distribution of was: 17.5% of current smokers (“yes”), 82.5% of current nonsmokers (“no”) and 0.8% of missing values. Step 1: a gradient boosting binary classification model was fitted to predict from covariates classically available in claims data (age, gender, medical diagnoses, and medication). In step 1, the dataset was considered as “external data”. Step 2: a variable was derived from ; all “no” values were set as missing and an increasing proportion (p) of “yes” were randomly set as missing. In step 2, represents what would have been captured in “claims data”. Step 3: was imputed using two approaches: (1) using the predictive model; (2) imputing “no” to all missing values (naïve approach) to gauge the added value of approach (1). For the two approaches and each p, the sensitivity (Se), specificity (Sp) and the accuracy of imputation were calculated. We hypothesized that approach (1) would lead to better accuracy than approach (2).

Results: For the fitted predictive model, Se=0.19; Sp=0.98 and AUC=0.77. For p=10% of “yes” replaced by missing (and all “no” being missing), the accuracy of approaches (1) and (2) were 0.97 and 0.98, respectively; for p=50%, the accuracy of approaches (1) and (2) were both 0.91; for p=90%, accuracy of approaches (1) and (2) were 0.85 and 0.84, respectively.

Conclusions: When the proportion of missing data in true smokers is substantial, predictive modelling trained on external data may add some value to a naïve approach by reducing misclassification of smoking=”yes” in true smokers.