(BSPOT49) Quality of and methodological heterogeneity in COVID-19 Vaccine safety studies – Case study of the myocarditis safety signal

Background: The risk of myocarditis associated with COVID-19 vaccines has been broadly investigated, yielding a wide range of effect estimates. It is unclear how much of this variation can be explained by design choices and methodological quality.

Objectives: To assess the methodological quality of published observational studies evaluating risk of myocarditis after COVID-19 vaccination, and to characterize the heterogeneity across studies addressing similar questions.

Methods: We conducted systematic searches on PubMed and EMBASE for observational studies reporting relative risks for myo-/pericarditis after COVID-19 vaccination published between Dec 11 2022, and Oct 9 2023. Signal detection and signal strengthening studies were excluded. Information on study methods and effect estimates was extracted. We assessed risk of bias (RoB) using ROBINS-I, adapted with additional signalling questions to accommodate specific features of self-controlled designs. We performed meta-analysis to quantify heterogeneity in effect estimates of studies addressing similar population–intervention–comparator–outcome (PICO).

Results: We included 24 papers, comprising 27 study designs and 49 different primary PICOs. Cohort (n=14, 52%) and self-controlled case series (SCCS) (n=9, 33%) were predominant designs. The main points of variation between studies were risk window length (7-266 days after vaccination; 41% used 28 days), timing of the reference period (33% before and after vaccination, 19% before only, 7% after only), control for important confounders (22% lacked comorbidities, 11% seasonality), and handling of COVID-19 infections (37% ignored them, 26% censored them, 15% performed sensitivity analyses 11% excluded them, 11% included in multivariable regression,). Effect estimates did not differ significantly between studies with high RoB (n=8, 27%) and lower RoB. We observed strong heterogeneity between studies on general population (P) for dose 2 of mRNA-1273 and BNT162b2 (I) compared to unvaccinated individuals/reference time (C) (I2=93%, point estimate range 2.63-56.5; I2=87%, point estimate range 0.62 -29.6, respectively). Among 25 effects estimated in the same populations by the same authors, using both cohort and SCCS designs, 76% of the times SCCS gave higher estimates than cohort.

Conclusions: We observed considerable variation in design choices such as risk window length, reference period timing, and control for confounding and COVID-19 disease. While the overall RoB did not seem to play a crucial role in effect estimate heterogeneity, the impact of other design choices warrants further investigation. This could inform evidence synthesis and guide future study design selection for vaccine safety.