(C-131) Conformal Prediction Framework and Selective Classification for Establishing the Credibility of Machine Learning Models – Application in Claims Data

Background: In most applications of machine learning (ML) it is critical to understand the level of confidence of model predictions. Conformal prediction is a paradigm that provides a statistically rigorous way to quantify uncertainty around predictions, offering safeguards, and supporting risk management. A conformal prediction framework for claims data could potentially provide a way to assess whether a model is fit for purpose.

Objectives: To present the conformal prediction framework and its potential in building trust in ML prediction models; to describe an application for the prediction of diagnosis and treatments in claims data.

Methods: We used a large claims database, the Optum’s de-identified Clinformatics® Data Mart Database, SQL for database queries, and Python for modeling.
We applied the framework for a use case of several rare sleep conditions: narcolepsy type 1 (NT1) and type 2 (NT2), or Idiopathic hypersomnia (IH). The models predicted the most recent diagnosis, and treatment classes.
Features from the longitudinal patient data included demographics, pharmacy and medical claims, procedures, and lab orders (all available data up to each predicted diagnosis/treatment).
Base classifiers included KNN, decision tree, support vector classifiers, and various ensemble methods including gradient boosting. Conformal scores then computed for each model, serving as a basis for the conformal prediction procedure. This produces a set of sleep disorder diagnoses and/or treatments with a definitive level of confidence and a bounded degree of uncertainty.
Selective classification was used to compare the outcome against a predefined threshold and abstain from classification for the samples whose predicted class probabilities were below the threshold, thereby ensuring minimal accuracy.

Results: Models accuracy improved by inclusion of both medical and pharmacy data but not lab orders. The XGBoost Classifier (XGBoost library), performed better than the other classification algorithms.
For about half the cohort, it was possible to rule-out one of the three sleep disorders based on medical claims. The model abstained at a critical confidence value between 0.55- 0.6. For small sets, the average level of accuracy was close to 100% and enabled accurate prediction for a sub-group of NT2. I.e., even when the base classifier provided limited accuracy, selective prediction could establish highly accurate predictive outcomes for subsets of the cohort.

Conclusions: The conformal prediction framework can help in establishing fit-for-purpose ML models by enabling the estimation of uncertainty and setting the desired level of confidence.