4E-04 - Bigger is better: replicating a signal detection study using tree-based scan statistics in two real world data sources

Background: TreeScan®, a tool used by FDA Sentinel for signal detection in real world data (RWD), uses tree-based scan statistics to detect an excess occurrence of un-prespecified, hierarchically structured health outcomes of interest (HOIs). Sentinel data are internal, and the ability to detect similar signals in other data sources is not known.

Objectives: To replicate Sentinel’s TreeScan signal identification analysis for erenumab (ERE), the first such analysis to use a self-controlled design, with two alternative RWD sources

Methods: We obtained the ERE analysis specification files and SAS code from the Sentinel study repository. We applied the Sentinel code to administrative claims data from Optum Clinformatics® and Komodo Health databases to identify incident ERE users from May 2018 to Jun 2022. Notably, Optum is part of Sentinel, and there may be additional overlap with the Komodo data. Each index date for an ERE episode was the first claim for ERE following a washout period. Inclusion criteria were age ≥18 years, no recent use of specific migraine treatments, and ≥456 days of continuous enrollment prior to the index date. Incident HOIs were those with no diagnosis code with the same first 3 ICD-10-CM digits in the 400 days prior. In Scenario 1, a tree-temporal scan was used to detect excess incidence of HOIs for each time interval within 183 days post-index. For Scenario 2, a tree-only scan was used to compare the incidence rates of HOIs for two time periods: 56-29 days prior to index and 1-28 days following index. Each scenario was conducted first with inpatient and emergency department diagnoses (Scenarios 1a and 2a) and second with the addition of outpatient diagnoses (Scenarios 1b and 2b). Incidence rate ratios with a p-value <0.05 were flagged.

Results: For Scenario 1, there were 67,058, 14,380, and 176,747 ERE episodes in the Sentinel, Optum, and Komodo data, respectively. The Sentinel population was older on average than Optum and Komodo populations (22.4%, 16.6% and 4.6% were aged ≥ 65 years, respectively). No HOIs were flagged in the Optum data for any of the 4 scenarios. For Scenario 1a, no HOIs were flagged in any data source. For Scenario 1b, 4 HOIs were flagged in Komodo out of Sentinel’s 5 HOIs, plus an additional 5 HOIs were flagged that were not in Sentinel. Sentinel was the lone data source with HOIs flagged for Scenario 2a. For Scenario 2b, 1 HOI was flagged in both Sentinel and Komodo, plus another 2 HOIs in Komodo that were not in Sentinel.

Conclusions: Our analysis suggests that data mining procedures such as TreeScan require very large sample sizes to effectively flag HOIs, and that flagged HOIs may vary by data source. Further research is needed to understand the performance characteristics of TreeScan under various scenarios.