(ASPOT12) External validation using real world data of a generative AI assisted medication-indication knowledge base

Background: Using the GPT-4 large language model (LLM), we previously developed and clinically validated a medication-indication knowledge base containing 10,853 medication-indication pairs (full details are described in an accompanying ICPE abstract). In order to assess the suitability of this knowledge base to support pharmacoepidemiology research, we aimed to validate this knowledge base using data from The National Health and Nutrition Examination Survey (NHANES), a large real world dataset including information on prescription medications and patient reported indications.

Objectives: To externally validate the LLM assisted medication-indication knowledge base using the NHANES database.

Methods: The prescription medications dataset from NHANES (pre-pandemic 2017-2020) was used. This contains patient-reported use of prescription medications, including the medication name, the ICD-10-CM code and patient-reported indication for each medication. Approximate string matching (fuzzy matching) was used to match medication between NHANES and the knowledge database and a random sample of 250 unique medication-indication pairs from the NHANES dataset were selected for external validation. Manual checking was conducted by clinicians to determine whether these pairs were captured in the medication-indication knowledge base. For pairs omitted from the knowledge base, desk research was carried out to assess potential reasons for omissions.

Results: Of the 2,899 unique medication-indication pairs in NHANES that were matched with the relevant medicine in the LLM assisted knowledge base, 250 (8.6%) were randomly selected for validation. Of these, 210 were captured by the knowledge base (overall agreement of 84.0%). Of the 40 indications not captured by the knowledge base, 7 (17.5%) were off-license indications missed by the knowledge base, 31 (77.5%) were incorrectly reported indications based on clinical review and 2 (5.0%) were valid indications missing from the knowledge base.

Conclusions: The reasonably high agreement between the medication-indication pairs in the knowledge base and the patient-reported indications in NHANES suggest that the former largely captures real-world prescribing patterns. Discrepancies were mostly attributable to potential misreporting of indications by respondents in NHANES, rather than being omitted by the knowledge base.