Using simulations to evaluate characteristics of negative control outcomes and their impact on bias detection

Background: Negative controls have been used as a powerful tool to detect and adjust for unmeasured confounding bias in observational studies. While most effort has been put toward ensuring that the selected negative control shares a similar path for bias, less attention has been given to the evaluation of the ability of the selected negative control to detect the bias. In fact, if a negative control fails to detect bias, it does not necessarily rule out the presence of bias in the exposure-outcome association—particularly, when the magnitude of bias is small, outcome event is rare, or sample size is small.

Objectives: To evaluate the characteristics of negative control outcomes (NCO) in detecting unmeasured confounding bias in a simulation study.

Methods: We conducted a simulation study involving datasets containing covariates, confounders, and a binary exposure in the context of a cohort study with 10,000 individuals. We simulated ten measured confounders and one unmeasured confounder. For demonstration, we generated two outcome models: a simple linear regression model (assuming a continuous outcome) and a generalized linear model (GLM) with log link and binomial family (a binary outcome). A three-way analysis estimated the probability of failure to detect bias (false negative, FN) by an NCO based on: (1) varying magnitude of bias due to unmeasured confounding, (2) varying ratio of the unmeasured confounding effect on NCO over the primary outcome, and (3) varying variance of the unmeasured confounder. All simulations and analyses were performed in R statistical program.

Results: In the simulation with a linear regression model, FN decreased as the magnitude of bias increased. When the ratio of the unmeasured confounding effect on NCO over the primary outcome decreased from 2 to 0.5, FN increased: from 12% to 93% when 10% of the observed exposure-outcome association was due to the unmeasured confounder, from 3% to 92% when 50% was due to the unmeasured confounder, and from < 1% to 92% when 75% was due to the unmeasured confounder. As the variance of the unexplained error term relative to the variance of unmeasured confounder increased, FN increased significantly. Similar trends were observed in a GLM outcome model with a binary outcome, albeit with differing degree of magnitudes.

Conclusions: When the magnitude of the unmeasured confounding bias decreases, relative effects of the unmeasured confounder on NCO vs. the primary outcome decrease, and as the variance of unexplained error relative to the variance of unmeasured confounder increases, the probability of FN increases. Including an analysis as illustrated in our simulation study, may help inform the reliability of the selected negative control for bias detection in observational studies.