Benchmarking Large Language Model Performance on Natural Language Processing Tasks for Pharmacoepidemiology

Background: The ability of large language models (LLMs) to interpret and generate human-like text has been accompanied with speculation about their application in pharmacoepidemiology. There is limited data available to inform evidence-based decisions on the appropriateness for specific use cases.

Objectives: We evaluated and compared four general-purpose LLMs (GPT-4, GPT-3.5-turbo, Flan-T5-XXL, and Zephyr -7B) and a healthcare-specific LLM (MedLLaMA) on five pharmacoepidemiology-related natural language processing tasks. The impact of systematic prompting strategies between models was evaluated.

Methods: We report results from evaluating each of the 5 LLMs on a subset of the Biomedical Language Understanding and Reasoning Benchmark (BLURB) tasks relevant to pharmacoepidemiology: identification of drug names (BC5-Chem) and diseases (BC5-Disease; NCBI Disease) from free text (i.e., named entity recognition [NER]), structured information extraction for literature reviews (PICO), and question answering (BioASQ). All models were evaluated without modification. Model performance was assessed according to a range of prompting strategies (defined by a systematic and reusable prompt engineering framework). Standard task-specific metrics defined by the BLURB benchmark were used: F1 (i.e., the harmonic mean of sensitivity and positive predictive value) for NER tasks, macro F1 for PICO, and accuracy for BioASQ.

Results: Across all tasks, GPT-4 outperformed other LLMs (average F1 for NER: 0.67; macro F1 for PICO: 0.32; accuracy for BioASQ: 83%). The most performant prompt for GPT-4 achieved over 85% accuracy for BioASQ, approaching previously-reported best results for models specifically created for the task. Depending on task, the second highest level of performance was achieved by Flan-T5-XXL (average F1 for NER: 0.53; macro F1 for PICO: 0.27) or GPT-3.5-turbo (accuracy for BioASQ: 78%). The domain-specific MedLLaMA model achieved lower scores for both NER and PICO, but outperformed Flan-T5-XXL on BioASQ, with an accuracy of 67%. We observed a substantial impact of strategically editing the prompt describing the task; for NCBI Disease tasks, the absolute difference in F1 between the best and worst prompt was 0.06 for GPT-4.

Conclusions: These results provide evidence of the potential LLMs may have for pharmacoepidemiology use cases. Even without medical or healthcare adaptation, GPT-4 had consistently robust performance. Continuing to explore how these emerging technologies can be fine-tuned for the healthcare setting, paired with human expertise, and enhanced through quality control measures will be important research to allow responsible innovation with LLMs in pharmacoepidemiology.