6E-04 - Comparison of Disease Risk Score Methods to Study Treatment Effect Heterogeneity in Real World Data: A Simulation Study

Background: Estimating treatment effects across levels of predicted baseline disease risk (i.e., the disease risk score (DRS)) is a popular method for evaluating treatment effect heterogeneity in randomized trials. However, the optimal approach for extending this method to real-world data studies is unclear, as investigators must avoid bias both from potential overfitting of the DRS model and from confounding due to baseline risk differences between treated and comparator groups. We propose an inverse probability-weighted, split-sample approach for training DRS models for the purposes of estimating treatment effect heterogeneity.

Objectives: The objective of this research is to evaluate bias in treatment effect estimates within strata of DRS when using the weighted split-sample method versus a previously proposed propensity score matching approach.

Methods: We simulated studies of a binary treatment by repeatedly sampling 3,600 individuals from a million-person population with 12 binary covariates and a confounded treatment-outcome relationship. We evaluated simulated treatment effects with corresponding odds ratio (OR) 1, 0.8, 0.5, and 0.5 with treatment-covariate interactions on the OR scale. In each scenario, we fit a DRS model on 1) an inverse probability-weighted random sample of 50% of comparators, who were then removed from the sample for treatment effect estimation (split-sample (SS) method), and 2) a 1:1 propensity score matched sample of treated and comparator subjects (matched-sample (MS) method). We estimated treatment effects within strata of predicted DRS on the risk difference scale. We calculated within-strata percent bias and overall percent bias (OPB), defined as the average absolute percent bias across strata. We repeated simulations varying outcome prevalence, study sample size, and comparator-to-treated ratio.

Results: In the MS method, bias increased as treatment effect increased and with treatment-covariate interactions (OPB 4.9% for OR = 0.8; 17.5% for OR = 0.5 with treatment-covariate interactions). In contrast, bias of the SS method was small across these scenarios (OPB 6.6% for OR = 0.8; 7.7% for OR = 0.5 with interactions).

In scenarios with treatment effect OR = 0.5 including covariate interactions, OPB decreased for both methods as sample size increased from 1,000 to 10,000, with a larger reduction for SS (-80%) than MS (-26%). OPB also decreased for both methods as outcome prevalence rose from 15% to 25%, with a greater decrease for SS (-59%) than MS (-35%). As the comparator-to-treated ratio increased from 1:1 to 4:1, bias decreased for SS (-64%) but increased for MS (+12%).

Conclusions: These findings support the use of weighted, split-sample methods to fit DRS models for studying treatment effect heterogeneity in real-world data.