Enhancing Drug Safety Review with Advanced Artificial Intelligence: A Pilot Study on Large Language Model-Assisted Literature Screening

Background: Drug safety review typically involves the integration of large amounts of text data from diverse sources. With the advancements in the field of Natural language Processing (NLP), particularly with the introduction of transformer architecture, the latest large language models (LLM) have showcased their capability to conduct general-purpose language processing, providing promising solutions to facilitate drugs safety review, including medical literature review.

Objectives: We aim to explore the potential of LLMs for improved efficiency of drug safety review. The pilot study is designed to evaluate the performance of LLMs for medical literature screening tasks.

Methods: We selected two epidemiology literature reviews on drug safety that were previously completed by FDA reviewers. The current approach of literature screening requires the reviewers to manually screen the query results from one or several literature databases (e.g., PubMed) to identify eligible studies for further review. In this pilot, we applied the state-of-art LLMs to perform this task and compared the results with the reviewer’s gold-standard adjudication. Specifically, we asked the model to predict which studies were eligible for further review based on the abstracts (>150 words). We selected a closed-source model (i.e., GPT-3.5) and an open-source model (i.e., Llama 2-70B) for testing. We applied different prompts with information on drug, safety outcome and study design to evaluate the stability of model predictions and the impact of several key words. For example, we compared the word “evaluate” versus “report”, and “association” versus “in relation to” in the prompts.

Results: Both LLMs successfully identified all eligible studies in the two sample reviews (4 out of 91 studies in Case #1; and 3 out of 42 studies in Case #2), reaching a sensitivity of 100%. The sensitivity remained perfect after prompt modifications. Using the original prompt, GPT-3.5 made two false positive predictions in Case #1 (specificity: 97.7%), and one false positive prediction in Case #2 (specificity: 97.4%). Llama 2-70B made three false positive predictions in Case #1 (specificity: 96.6%), and zero false positive predictions in Case #2 (specificity: 100%). After prompt modifications, the performance of GPT-3.5 remained the same, while Llama 2 made one additional false positive prediction in Case #1 (specificity: 95.4%).

Conclusions: Our pilot study demonstrated the tremendous potential of LLMs in performing literature screening tasks. We are evaluating their performance in additional datasets to study their robustness. This innovation can be extended to various other pharmacovigilance tasks.