(251) Comprehensive evaluation of algorithm demonstrates robust performance identifying pregnancy cases in VigiBase

Background: Especially robust post-marketing surveillance is necessary to assess risks of medicines and vaccines in pregnancy, as pregnant individuals are generally excluded from pre-marketing clinical trials. However, non-harmonised data processing and exchange for pregnancy exposures complicates the identification of relevant case reports in pharmacovigilance (PV) databases, especially in global databases. Thus, a rule-based algorithm was developed based on expert knowledge, inspired by similar ongoing work at European Medicines Agency, to identify pregnancy cases in VigiBase, the WHO global database of adverse event reports. The algorithm is available in VigiLyze, the VigiBase analysis tool available to national PV centres.

Objectives: To thoroughly evaluate an algorithm identifying pregnancy cases in VigiBase.

Methods: The algorithm utilizes structured report data elements such as reaction, indication, route of administration, gestational age, and seriousness, to identify reports of maternal exposure to a medical product before or during pregnancy, with or without adverse events in the pregnant person or the prenatally exposed. It does not flag patients over 50 years, paternal exposures, or lactation reports.
For evaluation, two datasets were created using de-duplicated VigiBase data up to Jan 2023. The dataset to estimate recall used a random subset of VigiBase, applying initial exclusion criteria (males over 18 years and females over 45 years) to allow for the low proportion of pregnancy reports. The dataset to estimate precision was created applying the algorithm to a straight random sample of VigiBase. To create a gold standard, four PV scientists annotated the cases in both datasets as pregnancy cases or non-cases. Inter-annotator agreement (IAA) using Fleiss’ kappa on a 297 report overlap between the annotators was calculated.
We compared the performance to a baseline defined by reports with reactions in the MedDRA SMQ ‘Pregnancy and neonatal topics’ (narrow).

Results: In the recall dataset with 7,874 reports, 253 were annotated as pregnancy cases. The algorithm flagged 190 of these, yielding a recall of 75%, 95% CI [69%, 80%]. Among the 63 cases missed by the algorithm, 48 were not reported in E2B. Restricting to E2B reports increased the recall to 91% [86%, 95%].
Among the 344 reports flagged by the algorithm in the precision dataset, 316 were annotated as pregnancy cases, yielding a precision of 92% [88%, 95%].
The baseline method had a recall of 62% [56%, 68%] and precision of 74% [69%, 78%], in the respective datasets. IAA was 0.83 indicating almost perfect agreement.

Conclusions: The pregnancy algorithm in VigiBase demonstrates high precision and high but reporting format-dependent recall, facilitating PV relating to pregnancy exposures in global data.