(174) Comparing publicly available methods for assessing covariate balance using standardized differences after implementation of weighting strategies

Background: When assessing conditional exchangeability between comparator groups, standardized differences can provide a numeric summary of balance in covariate distribution before and after implementation of weighting, matching or restriction approaches. Publicly available macros and statistical packages can be used to generate these summary measures, however, to whether calculation of these statistics differs across software or weighting strategies is unclear.

Objectives: We aimed to assess for differences in the calculation of standardized differences across publicly available methods when using weighting approaches.

Methods: Between 2016-2019, we identified a cohort of new users of Dipeptidyl peptidase-4 inhibitors (DPP4), aged >66, in a 20% random sample of Medicare fee-for-service beneficiaries (target population N=84879). A separate cohort of DPP4i initiators were identified in a sample of Medicare beneficiaries linked to a large medical system in North Carolina (NC) over the same period (sample population N=5514). We compared differences in outpatient visits (coded as a continuous covariate) during 1 year look back before and after weighting. We balanced the distribution of age/sex in the NC sample to the target population using standardized mortality ratio weighting (SMRW) and then the balanced age/sex of each group to the combined cohort using inverse probability of treatment weights (IPTW). Both stabilized and unstabilized were implemented. Standardized differences were calculated using the %STDDIFF SAS macro and the R ‘cobalt’ package in the crude and across all four weighting strategies.

Results: The crude standardized difference for outpatient visits was 0.225 across both methods. The %STDDIFF macro produced standardized mean differences of 0.075 and 0.078 for unstabilized IPTW and SMRW, respectively, compared to stabilized weights (0.227 both). The apparent improvement observed for the unstabilized weights were inconsistent with the weighted means which indicated no change. The variance calculation in the %STDDIFF macro when using unstabilized weights resulted in large increases in the variance estimates resulting in the observed decrease. Using cobalt, the standardized difference for outpatient visits was the same across all four weighting strategies (0.227).

Conclusions: Calculation of standardized differences in weighted study populations requires that estimation of variance accounts for large weights. Researchers employing the %STDDIFF macro should ensure the correct variance is specified or limit its use to settings in which the weights have been stabilized. This has implications for the calculation and interpretation of covariate balance metrics when conducting real world evidence studies.