(B-212) Adaptive multi-wave sampling for efficient chart validation

Background: There is increasing use of claims and electronic health record (EHR) data to evaluate the safety and effectiveness of medical products. For such studies, computable phenotypes are used to characterize patients and identify outcomes. Chart review studies establish reference labels against which computable phenotypes are compared to understand their measurement characteristics. However, establishing labeled charts is expensive and time consuming as medical experts need to review and label each chart. Approaches to minimize the number charts that need to be labeled are therefore needed.

Objectives: The first objective is to evaluate a quantity of interest (such as the positive predictive value, PPV) over sequential samples of charts, with the goal to minimize the number of charts reviewed by meeting some prespecified stopping threshold for success or futility. The second objective is to predict the stopping time (defined as the number of charts being reviewed) at which the chart review would be complete.

Methods: We develop a method to adaptively evaluate a quantity of interest (such as the PPV) over sequential samples of charts, with the goal to minimize the number of charts being reviewed. With the help of a simultaneous confidence band, we stop the chart review once the confidence band meets a prespecified stopping threshold (for instance, indicating futility). The developed approach leverages both different sampling techniques (random and stratified random sampling, Neyman’s sampling) and different confidence bounds on the quantity of interest (frequentist confidence bands and Bayesian credible intervals).

Results: Using a dataset of 6936 frailty score measurements in Medicare enrollees with liver disease, we observe that Bayesian credible intervals proved to be tighter than its frequentist counterparts. Moreover, we observe that simple random sampling often performs similarly to Neyman’s sampling. We demonstrate that it is possible to adaptively predict the stopping time of the chart review, with the prediction becoming more accurate as the chart review progresses.

Conclusions: We conclude that both random sampling and Neyman's sampling, in connection with Bayesian credible intervals, allows for an efficient validation of a response of interest (compared to the approach that validates all charts).