(154) Fit-for-purpose evaluation of Real-World Data for vaccines and medicines safety studies: the INSIGHT tool

Background: Existing fit-for-purpose (FFP) assessment recommendations, such as the Structured Process to Identify FFP Data, suggest defining study design elements necessary to conduct a study, by including data elements for valid capture, quality dimensions and ranking of importance. The European Medicines Agency (EMA) has developed a data quality framework (DQF), that evaluates these data elements through quality dimensions. Similarly, INSIGHT, an open-source tool, identifies potential data quality issues on ConcePTION common data model (CDM) datasets, used in ConcePTION, VAC4EU, and EU PE&PV Network.

Objectives: To describe the utility of INSIGHT as a FFP assessment tool.

Methods: INSIGHT was designed to assess data quality through 3 levels, generating 7 reports. Level 1 verifies conformance to CDM and evaluates completeness, level 2 logical relations and consistency. Level 3 is designed for evaluating trends over time and FPP assessment, comprising 249 configurable quality checks that can be used to operationalize study design elements for FFP assessment. Within a metadata file, 7 main categories of indicators can be pre-specified for FFP analysis: Study Parameters (start-end study dates, population distribution, age min-max, lookback period), Medicines, Vaccines (counts by ATC codes/vaccine type), Diagnoses (counts and incidence rates of first and/or recurrent event), Pregnancy (age min-max, provenance), Populations of Interest, and Masking categories to accommodate privacy restrictions at data site. INSIGHT was used to assess FFP of the data sources to be selected in the Covid-19 Vaccines Monitoring study (EUPAS42467), focusing on adverse events of special interest (AESI) linked to COVID-19 vaccine exposure.

Results: A FFP assessment was performed using Level 3 reports, linking to the dimensions of EMA's DQF (in parenthesis) by evaluating the minimum criteria for valid capture. Exposure COVID-19 Vaccines: presence of ATC codes/vaccine type (completeness), level of ATC (precision), counts of records (extensiveness) and incidence rates of uptake (accuracy), were checked in the Vaccines report. Outcome: Presence of AESI (relevance), number of cases (extensiveness) and incidence rates (accuracy) were checked in the Diagnoses report. Covariates: records of comedication (extensiveness), identification of comedications ATC codes (completeness) were checked in the Medicines report. Other study design elements were evaluated in similar fashion.

Conclusions: INSIGHT aids in FFP evaluation of Real-World Data sources by facilitating the visualization and deep assessment of study design elements for study specific determinants. INSIGHT quality checks can be seamlessly correlated with the EMA DQF.