1G-04 - The Relationship Between Sample Size and Standardized Mean Differences for Variable Selection: a Simulation Study

Background: In pharmacoepidemiology, analyses are often informed by quantitative comparisons of covariate distributions between those who received treatment and those who did not. One common metric for balance is the standardized mean difference (SMD), which is the difference in the proportion (or mean) of a covariate comparing treated versus untreated observations divided by the pooled standard deviation. While sample size is not directly included in the SMD formula, prior work has observed the SMD may be sensitive to the number of observations available.

Objectives: Compare performance of the SMD and p-value for variable selection in a relatively large versus small sample size using Monte Carlo simulation.

Methods: We created 1,000 datasets with 1) 10,000 observations and then 2) 500 observations. The binary exposure was created by random draws from a Bernoulli distribution with probability=0.5, as were 12 binary covariates with probability ranging from 0.07 to 0.95. In simulation, none of the covariates were confounders. Exposure and all covariates were used to predict the probability of the binary outcome. In each replicate, we calculated the SMD and the p-value based on a Chi squared test statistic for each exposure-covariate association. We calculated the crude risk ratio and risk difference and used generalized linear models to calculate the effects adjusted for covariates with SMD>0.1.

Results: In the first scenario (n=10,000), the average probability of the outcome was 12.9% (19.4% among exposed; 6.5% among unexposed). SMDs ranged from -.087 to 0.087. Across replicates, 0/12,000 SMDs were >|0.1|. As expected, 627 p-values (5.2%) were < 0.05. In the second scenario (n=500), 621 p-values (5.2%) were < 0.05. However, the SMDs ranged from -0.36 to 0.38, with 3,243 (27%) >|0.1|. After covariate adjustment, confidence interval coverage of the true effect estimate remained similar, but 64/1,000 (6%) of the log binomial risk regression models did not converge.

Conclusions: Compared to using the p-value to detect covariate imbalance, the proportion of imbalance identified using the threshold of SMD>|0.1| varied depending on total sample size. There were chance imbalances in nearly 1/3 of covariates across the replicates when n=500. In smaller samples, consideration of graphical models, the clinical relevance of covariates that exceed the SMD cutoff, and measures of uncertainty around the SMD values themselves may be warranted, especially when using the SMD to make modeling choices.