3D-01 - Extracting adverse event timelines, severity, and resulting interventions from clinical notes of patients receiving CAR-T therapy using large language models.

Background: Chimeric Antigen Receptor T (CAR-T) therapy is associated with risk of severe adverse events. These details are usually unavailable in structured data, making manual clinical notes review time-consuming and often impractical. Recent advances in large language models (LLMs) show potential for extracting information from clinical text.

Objectives: We aimed to extract patient adverse events post-CAR-T administration, focusing on both a predefined list of expected adverse events and those not on this list. We also aimed to extract the severity grade for Cytokine Release Syndrome (CRS) and Immune Effector Cell-Associated Neurotoxicity Syndrome (ICANS).

Methods: Patients were included in a cohort study at the administration of any of the six available CAR-T. We extracted progress notes within 30 days of the CAR-T administration from UCSF’s deidentified clinical data warehouse. Using GPT-4 via a HIPAA-compliant Conversion Application Programming Interface (CAPI), we extracted CAR-T adverse events through four prompts per note: 1) to identify prespecified adverse events, 2) to identify non-prespecified events, 3) to identify CRS grade, and 4) to identify ICANS grade. All prompts also requested extraction of clinical interventions addressing the adverse events. One oncologists (AS) and one pharmacist (JG) evaluated a random sample of adverse events from random notes to evaluate LLM performance.

Results: We identified 4,762 clinical notes within 30 days of CAR-T administration from 293 patients (40.0% women, 60.0% men; median age 65.6 years [IQR: 54.8 – 71.7]).

Among performance metrics, manual validation of a random-note sample showed GPT-4 extracted prespecified adverse events with 70% accuracy and their clinical interventions with 70% accuracy; non-prespecified adverse events with 70% and their clinical intervention 85%; CRS grades with 95% accuracy and their clinical interventions with 90% accuracy; ICANS grades with 90% accuracy and their clinical interventions with 90% accuracy

The three most frequent prespecified adverse events were CRS (80% of patients; median 4 days [2–7]), neutropenia (69%; 16 days [9–23]), and neutropenic fever (64%; 4 days [2–7]). The three most frequent non-prespecified adverse events were fever (4%; 5.75 days [4–9]), nausea (4%; 6 day [1–15.50]), and Hypotension (3%; 7.50 days [4.50–12.50]). CRS was most frequently a grade 1 event, representing 30% of patients with CRS records. ICANS was most frequently a grade 1 event, representing 15% of patients with ICANS records.

Conclusions: We were able to extract adverse events whose nature and timelines align with the literature using prompts developed through basic prompt engineering. This encourages further exploration of LLMs for extracting clinical events from unstructured clinical text and estimating their timelines and severity.