(208) Using large language models with real world data to support regulatory decision-making for medicinal products: Are we on the road to regulatory convergence or chaos?

Background: The application of large language models (LLM) in the context of non-interventional studies using real world data (RWD) offers enormous efficiencies and analytic capabilities. To date, regulatory guidance is still evolving regarding the use of LLMs and artificial intelligence (AI) in drug development. There has been no effort to compare and synthesize the commentaries to determine where they converge and where gaps remain.

Objectives: 1) To determine which agencies have issued position statements on the use of LLMs with RWD; 2) To critically appraise these documents for degree of convergence regarding: (a) governance, accountability, and transparency; (b) standards for data quality, reliability, and representativeness; and (c) model development, performance, monitoring, and validation; and 3) To identify any gaps.

Methods: We conducted a literature search to identify relevant position papers issued within the past 5 years on the uses of generative AI /LLMs in drug development, excluding congress abstracts and gray literature. Based on the research objectives, a data abstraction form was developed and piloted. Full-text abstraction was conducted.

Results: Our search identified 2 documents: 1) an EMA ‘Reflection Paper’ (2023), and 2) an FDA ‘Discussion Paper’ (2023). An in-depth comparison (to be presented tabularly) showed important differences. EMA sets forth ‘good practices’ for industry in using AI/ML in product development; FDA proposes general principles and standards for the application of AI/ML as the basis for ongoing dialogue with key stakeholders. Both advocate for the intentional use of AI/ML, and a proactive, risk-based approach with regard to both quality and regulatory concerns centered around human-led governance. Gaps common to both documents include little detail concerning the specific application of LLMs with RWD, and recommended methods for managing undesirable outputs (e.g., hallucinations).

Conclusions: Regulatory science regarding the use of LLMs in drug development is nascent; only two regulators have shared initial perspectives on this topic to date. The EMA and FDA commentaries vary in terms of intent (prescriptive vs. exploratory), and development approach (one-time stakeholder feedback versus ongoing collaboration). Commonalities are evident in regard to high level principles for oversight, data quality and model development and validation. A major gap is that both lack detail regarding the use of LLMs with RWD. As AI/LLMs is rapidly evolving, a key challenge all regulators will face is how to avoid ‘regulatory lag’. Harmonization across regulatory frameworks will be critical to ensure that its benefits can be fully realized.