(SLC-012) A Framework to Detect and Quantify Operator-Learning Effects in Medical Device Safety Evaluation

Background: Incremental risk due to procedural learning effects may be falsely attributed to the device, leading to discontinuation of effective treatments. Conversely, if intrinsic device safety risks are mistakenly attributed to learning, missed or delayed identification of such risks may harm patients.

Objectives: To develop and assess the statistical performance of an analytic framework to detect the presence of learning effects in medical device registry data and quantify the learning curve associated with the introduction of a novel medical device.

Methods:

Design: Simulation study using synthetic data representing real-world data complexity with known ground truth learning and safety signals. The analysis team was blinded from the data generation process.
Setting: We generated synthetic datasets based on observed clinical distributions and complex feature correlations among patients hospitalized at U.S. Department of Veterans Affairs facilities. Procedural learning effects, device safety signals, device prevalence, and outcome rates were varied based on prior device safety reporting.
Exposures and interventions: Simulated patient exposure to a novel device-of-interest or a reference device across multiple medical centers and clinical proceduralists with patient order in the assigned provider’s device-specific case series.
Main outcome measures: Sensitivity and specificity of correctly detecting the presence or absence of learning effects and root mean squared error between specified and estimated learning curves.
Statistical analysis: We fit generalized additive models to determine the presence of learning effects and to estimate learning curve functional forms and parameters when learning was present. Detection accuracy and estimated learning effects were evaluated against the specified values for each simulated dataset.

Results: Among 2,483 simulated datasets, the presence or absence of learning was correctly detected in 2,233 (sensitivity=90%; specificity=88%). When we were able to estimate learning curves, the root mean squared error between the estimated and specified curves was < 1 for 81% datasets, increasing to 96% when the cutoff was < 3. Our approach was most successful at detecting and quantifying learning when learning speed was slower, learning was larger in magnitude, and the novel device was more broadly used.

Conclusions: In medical device safety evaluation, operator-learning effects associated can be effectively modeled and characterized, allowing for the subsequent disentanglement of learning and intrinsic device safety risks. This work warrants subsequent framework validation using real-world clinical datasets.