(C-207) Implementing robust machine learning pipelines with real-world data: identifying predictors of immune mediated adverse events among patients with advanced melanoma treated with immune checkpoint inhibitors

Background: Predictive modeling with real-world data (RWD) is complicated by complex statistical relationships and large numbers of predictors. Machine learning (ML) algorithms handle such complexity and automate variable selection. However, ML is more than a fitted model but a pipeline of preprocessing and fitting steps. To enhance generalizability and ensure valid assessments, model specification, tuning, and validation must consider the entire pipeline.

Objectives: To demonstrate a robust ML pipeline in RWD to identify predictors of immune mediated adverse events (imAE) in patients with advanced melanoma receiving immune checkpoint inhibitors (ICI).

Methods: Patients ≥ 18 years old with advanced melanoma initiating first line ICI treatment between Jan 1, 2016 and July 31, 2023 were identified from the ConcertAI oncology dataset, which is comprised of medical records from US oncology practices. ImAE were identified from treatment start through death, treatment discontinuation, new line of therapy, or study period end. Baseline characteristics, diagnoses, and medications were abstracted into 318 variables. Preprocessing steps were specified for feature engineering, preliminary variable screening, and imputation, and these were included in the training and validation processes. The data were split into training and validation sets, and four models were compared: main effects Cox proportional hazards (ME-CPH), flexible penalized Cox proportional hazards (FLEX-CPH), random survival forest (RSF), and gradient boosted survival tree (GBST). Bayesian tuning with 10-fold cross-validation was performed in the training set to optimize the area under the precision recall curve (AUPRC) at 6 and 12 months post-index. Performance was assessed in the validation set, and the best model was used to estimate variable importance with Shapley values. Areas under the receiver operating characteristic curve (AUROC) and Brier scores were computed.

Results: A total of 1489 patients were included in the study (median follow-up 4.8 months), of whom 189 (12.7%) experienced an imAE. After preprocessing, 131 candidate variables remained. The GBST performed best (AUPRC: 0.38), followed by FLEX-CPH (0.35), ME-CPH (0.35) and RSF (0.31). The AUROC and Brier scores for the GBST were 0.62 and 0.22, respectively. The top 3 predictors by Shapley value were ECOG score, potassium level, and number of metastases.

Conclusions: Our model had low to moderate performance, highlighting challenges of identifying advanced melanoma patients at high risk for imAE. Nonetheless, by utilizing a robust ML pipeline we identified both previously described predictors of imAE and additional variables that may warrant further investigation for their role in the development of imAE.