(A-029) Transparency in the Training, Validation, and Testing of Artificial Intelligence-Based Tools for Real-World Evidence Synthesis

Background: Artificial intelligence (AI)-based tools have shown promise in automating labor-intensive tasks in systematic reviews for pharmacoepidemiology (PE) outcomes research, and pharmacovigilance, particularly for screening literature search outputs and data extraction. However, concerns remain regarding the transparency of their training, validation, and testing methodologies, which are critical for ensuring their reliability and regulatory acceptance.

Objectives: To examine how available AI-based tools used for systematic reviews in PE and outcomes research are trained, validated, and tested.

Methods: A systematic review (PROSPERO, CRD pending) of proceedings from the conferences of ISPE, ISPOR, and ISoP was conducted, covering the period from January 1, 2019 to January 2, 2025, to identify systematic reviews supported by AI technology. Parameters of interest included the name of the AI tool, intended purpose (screening, data extraction, and/or methodological quality assessment), stage of development, training and validation methods, and characteristics of the test sets (type, sample size, and therapeutic area). Two independent reviewers performed abstract screening and data extraction.

Results: A total of 1,540 unique conference abstracts were identified, of which 96 were systematic reviews supported by AI technology (5 from ISPE, 87 from ISPOR, and 4 from ISoP). The majority of AI tools focused on title and abstract screening (75.9%), while data extraction accounted for 24.1%. Training methods were reported in 58.6% of abstracts, while validation (34.5%), testing (37.9%), and prompt engineering (37.9%) were less frequently documented. Sensitivity for source screening ranged from 27% to 100%, and specificity ranged from 8% to 100%.The reported proportion of references used for training sets ranged from 5% to 80%, with two sources indicating that a 20% training set yielded the best predictions or the greatest time savings.

Conclusions: Despite the growing use of AI tools for real-world evidence synthesis reviews, inconsistencies in reporting training, validation, and testing methodologies persist, potentially affecting the transparency and reliability of these tools. As annual conferences are key platforms for disseminating innovative methods, ensuring methodological transparency is essential to support broader adoption and regulatory confidence in AI-assisted systematic reviews.