Identifying adverse drug events in clinical notes of Intensive Care patients: the need for a gold standard corpus to enable automation via Natural Language Processing

Tuesday, August 27, 2024

1:45 PM – 2:00 PM CEST

Location: Conv Hall I D

Presenting Author(s)

Rachel M. Murphy, MPharm MSc (she/her/hers)

PhD Student
Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
Amsterdam, Netherlands

Slides

Co-Author(s)

DD

Dave A. Dongelmans

Department of Intensive Care Medicine, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands, Netherlands
Nd

Nicolette F. de Keizer

Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands, Netherlands
RJ

Rosa J. Jongeneel

Amsterdam University Medical Centers, Netherlands
CK

Christiaan H. Koster

Department of Anesthesiology, Amsterdam UMC, Vrije Universiteit, Amsterdam, the Netherlands, Netherlands
KJ

Kitty J. Jager

Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands, Netherlands
AA

Ameen Abu-Hanna

Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands, Netherlands
IC

Iacer Calixto

Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands, Netherlands
Joanna E. Klopotowska, PharmD, PhD (she/her/hers)

Assistant Professor, Principal Investigator
Department of Medical Informatics, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands
Amsterdam, Netherlands

Background: Adverse drug event (ADE) prevention in Intensive Care Unit (ICU) patients is hindered by a lack of up-to-date and detailed information on the occurrence of these harmful events. Natural Language Processing (NLP) is a type of machine learning that may contribute to the identification and extraction of information about ADEs from clinical notes documented in the Electronic Health Record (EHR). However, to capitalize on this NLP potential, we first need gold standard human-labelled clinical notes to train and validate NLP models.

Objectives: To create a gold standard corpus of labelled ICU clinical notes, suitable for training and validating NLP models for ADE detection in Dutch clinical notes, and to assess the value of clinical notes (unstructured data) for ADE detection in comparison to ICD-10 codes (structured data).

Methods: We used de-identified clinical notes pertaining to 102 adult ICU patients admitted to Amsterdam University Medical Centres, The Netherlands, over a four-year period (November 2015 – January 2020). The notes were selected based on previous work and each note contained at least one ADE. Two physician annotators labelled the notes to identify mentions of drugs and disorders, as well as the potential relationship between these entities indicating an ADE. The final gold standard corpus was a result of adjudication of the two sets of expert labels. To compare ADE yield from clinical notes to ICD-10 codes, we searched for ICD-10 codes documented for these patients which describe harm from drugs.

Results: We identified 261 ADEs in the 102 annotated clinical notes (median 2 ADEs per patient). Most notes (n=79, 77.5%) mentioned one, two, or three ADEs, with 11 being the highest number of ADEs in a single note. The name of the suspected drug or drug group was specified in 83.7% of the ADEs. Creation of the gold standard corpus took 541 hours. Using ICD-10 codes, we only identified one ADE in the included 102 admissions (T88.7, Unspecified adverse drug event). Analysis of the ICD-10 codes took 16 hours.

Conclusions: While creating a gold standard corpus is more time-consuming in comparison to using ICD-10 codes, the wealth of valuable information obtained about ADEs, both in terms of their number and the amount of detail provided, makes the effort worthwhile. Future work aims to use the created gold standard corpus to train NLP models to automate ADE information retrieval from a larger set of EHR notes, which may substantially reduce medical experts’ time.