(349) Patient-level prognostic models for COVID-19 vaccine effectiveness and safety: a systematic review

Background: A reliable model for predicting adverse events following vaccination might help reduce vaccine hesitance and prevent serious outcomes such as hospital admission for individuals at high risk. Likewise, a model that predicts vaccine effectiveness can advise further protection in case of breakthrough infection for individuals with poor immune response. Currently, there is no widely adopted prognostic model in clinical practice.

Objectives: To systematically review the quality of prognostic models developed for the safety and effectiveness of COVID-19 vaccines.

Methods: Systematic review. MEDLINE and EMBASE databases from 01/01/2020 to 04/05/2023. Articles reporting on the development, with or without external validation, of prognostic models for individualised predictions on COVID-19 vaccine safety or effectiveness were included in our review. We set no restrictions for vaccine brands. Data source, patients’ characteristics, outcomes to be predicted, setting, study population, sample size, number of individuals with the outcome, number of predictors, modeling technique, internal validation type, and predictive performance measures (discrimination and calibration) were among the extracted data. The Risk of Bias was evaluated using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Descriptive statistics and visual plots were employed for data presentation and analysis.

Results: We included 22 publications reporting 58 models (safety n=26/58, effectiveness n=32/58). Outcomes included death (n=12/58), hospital admission (n=11/58), and side effects (n=22/58). Most models were developed following administration of Pfizer vaccines (n=37/58). For safety models, data from self-reporting questionnaires was used (n=9/26). Despite the variety of predictors, the median count in model development was relatively low (7). Few studies (n=2/22) addressed changes in the infection rates during the study period, missing data, competing events or loss to follow-up/censoring. While most models reported high levels of discrimination (AUC>0.90), calibration was seldom reported. QCovid3 model by Hippisley-Cox et al. shows low risk of bias and may serve as a valuable guide for clinical decisions regarding vaccine effectiveness, in-house validation. It uses age, sex, ethnic origin, deprivation, body mass index, comorbidities, and infection rate.

Conclusions: Most studies on prognostic models on effectiveness and safety of COVID-19 vaccines show limited reporting and poor methodological quality and, consequently, are at high risk of bias. Given these constraints, predictive performance seems optimistic, contraindicating their use in clinical practice. Future models should address methodological gaps to ensure robust predictions.