(C-127) AI Trustworthiness: A Study on the Calibration of Large Language Models in Pharmacoepidemiology

Background: Large language models (LLMs) are increasingly used in pharmacoepidemiologic (PE) research for tasks like literature review, protocol development, and accessing unstructured data. However, their trustworthiness, i.e. the reliability of such models in providing accurate insights, remains largely unexamined.

Objectives: Quantitatively assess the trustworthiness of LLMs for PE applications by measuring calibration, i.e. the alignment between a model's reported confidence and its accuracy in practice.

Methods: We conducted a comprehensive evaluation of 9 LLMs—both open- and closed-source (OpenAI’s GPT 3.5 and 4)—using the Biomedical Language Understanding and Reasoning Benchmark (BLURB), a collection of over 28,000 samples, distributed across 13 biomedical tasks that are critical for PE, such as drug-event association extraction and medication identification. Calibration was assessed via the Expected Calibration Error (ECE) metric across 3 confidence assignment methods: verbal (self-assessment), self-consistency (similarity across repeated responses), and hybrid (aggregate of multiple self-assessments). We also evaluated 2 post-hoc calibration techniques that are traditionally employed to mitigate miscalibration—isotonic regression and histogram binning.

Results: Significant miscalibration was observed across all LLMs. Model-specific ECE ranged from 29.8% (±21.7) to 47.2% (±42.4); lower ECE is better. When grouping models (small [<=10B parameters] vs large [>10B parameters], and general-domain vs biomedical-specialized), the biomedical-specialized large model showed better calibration (36.1% ECE, ±23.3) particularly compared to general-domain small models (39.5% ECE, ±23.1). Task-specific performance varied substantially from 23.9% (±26.4) for sentence similarity (useful for literature summarization) to 46.6% ECE (±19.8) for relation extraction (crucial for drug-safety signal detection). Self-consistency emerged as the most reliable confidence assessment method (27.3% ECE, ±18.0). Post-hoc calibration improved ECE by 11.7-32.9%.

Conclusions: This study provides critical evidence that current LLMs require careful assessment before deployment in PE research. The high degree of expected error indicates that relying on out-of-the-box confidence scores from LLMs to impart reliability in practice is risky, as high error rates for tasks such as medication extraction and signal detection can put patients at risk if not evaluated. LLMs and confidence assignment strategies must be assessed and deployed judiciously, depending on the required task; validation is paramount.