(134) A review of recommendations in guidelines on missing data in prediction modelling

Background: Clinical prediction models are increasingly recognized as “software as a medical device” by the U.S. Food and Drug Administration (FDA). These are being integrated in hospitals’ electronic health records (EHR) as clinical decision-support tools for patient risk assessment and to inform treatment selection. Yet, there is little guidance about how to address missing data in the EHR when developing, validating, and implementing clinical prediction models.

Objectives: To identify guidance about how to address and report on missing data in prediction modelling studies.

Methods: We systematically searched PubMed, seven guidelines' websites, and the EQUATOR Network for published methodological guidelines about development, validation, or implementation of clinical prognostic or predictive models through June 14, 2024. We also searched for regulatory guidance on the websites of the European Medicines Agency, Health Canada, the U.S. FDA, and the International Council for Harmonization. Two authors independently searched and screened the literature, including full-text screening of articles, based on a pre-specified protocol. Disagreement was resolved by consensus. We extracted existing recommendations from the included literature and synthesized the results.

Results: From 1140 articles and 12 federal documents, we identified 18 articles and 5 regulatory files of relevance. Nine of the articles address the use of artificial intelligence (AI) for prediction. Two of the regulatory documents (FDA’s guidance about Clinical Decision Support software and Transparency for Machine Learning-Enabled Medical Devices) include recommendations regarding missing data. All recommendations from the literature are about the model development stage, including recommendations about reporting (14/18), making explicit assumptions about the missingness mechanism (5/18), and statistical methods for management of missingness (14/18). The TRIPOD statement and a framework for “end-to-end clinical AI” advise researchers to “define and handle” missing data and describe possible reasons for missingness. Multiple imputation is the most frequently recommended method, preferable to complete case analysis. Regulatory documents recommend investigators communicate information that could impact patient outcomes and consider what information is needed by the intended users. We found no specific recommendations for addressing missing data in the EHR during the model validation stage or when implementing models in practice.

Conclusions: With the growing interest by regulatory agencies and widespread inclusion of prediction models in clinical workflows, guidelines need more specificity regarding handling of missing data in validation and deployment of models in practice.