(177) Development and Clinical Validation of a Generative AI Assisted Medication-Indication Knowledge Base

Background: Information about the relationship between medications and clinical indications is used widely in biomedical research, but it is not readily available in a standardised format. Existing databases have limitations such as restricted scope (e.g. lack of coverage of off-label use) or are not accessible in a machine-readable format to enable real world evidence analytical workflows. The recent development of highly capable large language models (LLMs) presents an opportunity to develop a comprehensive medication-indication knowledge base.

Objectives: We aimed to create a medication-indication knowledge base using LLMs and conduct clinical validation of its precision.

Methods: Unique medication names were extracted from the English Prescribing Dataset, excluding categories such as food and nutrition products, vaccines, anaesthetics, appliances, and dressings. The GPT-4 LLM was used to generate medication-indication pairs after developing a prompt pipeline with fine tuning of the model parameters and testing against known medication-indication pairs. Results were clinically validated by three independent reviewers with medical or pharmacy training to determine the precision rate on samples stratified by a model generated confidence score for each medication-indication pair.

Results: A total of 1,540 unique chemical substances were identified and used for generating indications. The output produced 10,853 medication-indication pairs with associated confidence scores. Clinical validation was conducted on a stratified random sample by three independent reviewers on 645 unique pairs (5.9%), of which 301 pairs (2.8%) had a confidence score of at least 0.75 (where 1 = maximum confidence). 282 out of the 301 pairs were assessed to be correct, demonstrating a precision rate of 93.7%. Both licensed and off-label indications were included in the output. Hallucinations were observed with additional medication names and erroneous indications in the output, especially at low confidence scores.

Conclusions: Large language models represent an advancement in providing comprehensive and accurate medication-indication information to support real world evidence research. The large number of medication-indication pairs and high precision rate observed during clinical validation highlights LLM’s potential value in biomedical research.