(C-177) Power vs Precision When Estimating Risk Differences Using Observational Data? A Case Study from TARGET-IBD

Background: Real-world observational studies often use sample size and power calculations modeled after randomized clinical trials. Such approaches can be overly restrictive for causal analyses of real-world data where sample size is fixed and adjustments for confounding and informative censoring are necessary. In such settings, evaluating precision—the amount of random variation in the study estimates—may be more useful.

Objectives: To evaluate statistical power and precision for estimating the risk difference (RD) for surgery or hospitalization among patients with inflammatory bowel disease (IBD) who achieved disease remission earlier in their treatment vs. those who reached remission later.

Methods: Using data from TARGET-IBD, a U.S.-based IBD registry, we identified 316 adult ulcerative colitis (UC) and 478 Crohn’s Disease (CD) patients who initiated advanced therapy (i.e., small molecules or biologics). We consulted clinical experts to obtain plausible ranges for key input parameters [including early remission prevalence (p1) and baseline risk of surgery or hospitalization (r1)] required to compute power and precision for the known sample sizes and RD of 10%. To account for additional uncertainty resulting from adjustment for moderate confounding (odds ratios of ~1.95) via inverse probability of treatment weighting (IPTW), we calculated a variance inflation factor (VIF) (Austin, 2021) and used it to derive the effective sample size (ESS), assuming 10% annual loss to follow-up. Power was calculated based on the ESS using G*Power 3.1.9.6 (two-sided test, alpha=0.05). Precision was assessed by constructing 95% confidence intervals (CI) for the RD, adjusting for censoring and the VIF. Sufficient power and precision were defined as >80% power and 95% CIs excluding the null value of zero, respectively.

Results: Based on clinical input, we constructed n=6 scenarios in which p1 was 25-45% for UC and 32-52% for CD patients and r1 was 14% for UC and 48% for CD. Estimated VIFs ranged from 1.03-1.26 for UC and 1.06-1.21 for CD resulting in ESSs of 238-253 for UC and 372-384 for CD. Power ranged from 63.2%-78.8% for UC and 45.6-50.8% for CD. Although no scenario achieved sufficient power, n=5 (83%) achieved sufficient precision. For example, in one UC scenario, power was only 63.2%, yet the 95% CI (3.2%, 16.8%) excluded the null value.

Conclusions: Precision and power analyses yielded contradictory assessments of study feasibility. If we had only calculated power, we may not have proceeded with this study. Our findings demonstrate that secondary data can provide valuable insights even in studies with limited power. Precision—not power—should guide feasibility assessments in observational studies.