(258) Evaluation of the Vaccinated Concurrent Comparator Methodology for Near Real Time Safety Surveillance

Background: Near real time surveillance provides timely detection of potential post-market safety signals. When using a historical comparator, the design compares post-vaccination outcome incidence to aggregate historical incidence. Using a historical comparator has limited control for confounding factors, which may lead to biased results. Alternatively, the vaccinated concurrent comparator design aims to reduce bias by comparing outcome incidence in risk and calendar-matched comparator periods within a vaccinated population. There has not been a formal comparison of the performance of these two designs.

Objectives: To compare the performance metrics of the historical comparator and the vaccinated concurrent comparator designs for near real time surveillance using simulated data.

Methods: The number of outcomes on each day was simulated by a Poisson model. A total number of 864 simulation scenarios had several varied parameters including hypothetical vaccine uptake patterns, length of risk and comparator periods, magnitude of incident rate ratios (IRRs), and baseline outcome rates. Performance metrics were calculated for both designs, including: the power, false positive rate, time to signal, and mean square error (MSE). Analysis denominators differed based on specific metric comparisons.

Results: The two designs offered advantages over each other under different scenarios. Power of the two designs was similar for common outcomes with a baseline rate of 0.7 outcomes per 100,000 person-days, but for rare outcomes (0.007 outcomes per 100,000 person-days), the historical comparator design had higher power across all 360 scenarios (average power: 44% vs. 22%). The false positive rate was < 0.05 for both designs across all scenarios when the baseline outcome rates did not vary throughout the study period. However, if the baseline rate varied, the false positive rate for the historical comparator design was inflated in 56 of 96 scenarios (average false positive rate: 0.43 vs. 0.02). Time to signal for the concurrent comparator design was longer in 850 of 864 scenarios (average time to signal: 96 days vs. 34 days). MSE of the historical comparator design was lower than the concurrent comparator design in 227 of 288 scenarios when the baseline outcome rate did not vary in the study period (average MSE: 13.6 vs. 10.5).

Conclusions: The historical comparator and vaccinated concurrent comparator designs exhibit strengths and weaknesses in terms of performance in the context of near real time surveillance. When choosing between the two designs, decisions should be made based on the study-specific characteristics and priorities.