(ASPOT15) Automation of study report abstract generation by Large Language Models, methodology and application example

Background: Since 2022, ChatGPT by OpenAI and Large Language Models (LLMs) have been used in scientific and academic fields. It seems a promising tool to facilitate thorough literature reviews and summarizing scientific contents (e.g. scientific articles, patient records, etc.).

Objectives: To assess the feasibility of automatically generating clinical study report abstracts testing several LLMs methods.

Methods: A dataset of ten clinical study reports (CSR) was used. It was a representative sample of pharmaco-epidemiological studies conducted on primary or secondary data, with various study design and various objectives (description of patients, epidemiological assessment, survival studies, pharmacoeconomic evaluation).
We explored a full methodology to tackle the challenges of automatic summarizing. First, MapReduce was used to handle large documents (tens of pages). Then, an abstract template containing the sections to be respected was supplied to the LLM. For each pre-defined section instructions detailing the purpose and practical information (number of characters to be respected, whether it was necessary to transliterate certain parts without modifying the content, etc.) was specified.
The technical implementation was done in python 3.11 language on LangChain framework, using OpenAI’s ChatGPT (3.5 and 4) on Azure. Several versions of the LLM were tested, using either a single parser of the report content or iterative parsers in loops, using ChatGPT 3.5 vs 4, using different output formats. At all times, the model was hosted on a private instance (French tenant) for sovereignty and confidentiality purposes. The summarized content was produced using Retrieval Augmented Generation (RAG). A quality indicator for the generated abstract was calculated, by comparing it with a manually written abstract (at the time the report was drafted).

Results: The abstracts generated by each LLM method were compared to the abstracts written by a pharmacoepidemiologist. Each method was ranked according to quality indicators. Quality indicators included content relevance, processing time, respect of output format and direct costs of ChatGPT. This approach was summarized in a benchmark table of all models, to demonstrate the challenge of creating a model outperforming on all the criteria.
Limitations still stand in the way quality and accuracy of LLMs, including biases linked to the dataset used for training, lack of transparency in the process, the concept of hallucinations, referencing of non-existent citations, copyright issue, ethical issue.

Conclusions: LLMs appear to be promising tools for producing summaries of clinical contents allowing to focus on tasks requiring significant human intellectual input, thereby saving valuable time in the research process.