3D-05 - Pitfalls of Using AI Tools to Write Statistical Analysis Plans for Pharmacoepidemiology Target Trial Emulation in Electronic Health Insurance Databases Without Pre-Training

Background: Artificial intelligence (AI) tools are increasingly used to assist academic research. However, the potential pitfalls of using AI to generate statistical analysis plans (SAPs) for pharmacoepidemiology target trial emulation (TTE) using electronic health insurance databases remain unclear.

Objectives: To explore the pitfalls of using AI tools to write statistical analysis plans for pharmacoepidemiology target trial emulation in electronic health insurance databases without pre-training.

Methods: We referenced a research question from a published study, which aimed to estimate the relative effect of non-vitamin K antagonist oral anticoagulants (NOACs) versus warfarin on the risk of diabetes complications and mortality in atrial fibrillation (AF) patients with diabetes mellitus (DM) using the Taiwan National Health Insurance Research Database (NHIRD). This case study served as the basis for developing a study protocol. Nine research fellows trained in pharmacoepidemiology were invited to read the reference and draft text instructions for GPT to generate a corresponding SAP for the hypothetical causal question. The goal was to ensure that the generated SAP would enable them to replicate the study. Once the text instructions were developed and collected, they were randomly assigned to another fellow, who then used them as prompts on an AI platform (ChatGPT 4.0) to generate the SAP protocol. A group of epidemiologists evaluated the generated outputs based on three key dimensions: (1) knowledge of NHIRD, (2) accuracy of operational definitions for exposure and outcome variables, and (3) adherence to TTE design principles.

Results: Nine AI-generated SAPs were collected and evaluated. Three SAPs did not define the “index date” or “time zero.” None of the SAPs correctly described the structure of NHIRD. Four SAPs proposed using variables that were not available in NHIRD. Five SAPs failed to provide comprehensive diagnosis codes or Anatomical Therapeutic Chemical (ATC) codes to define exposure and outcome variables. Additionally, seven SAPs did not adhere to TTE design principles. Five SAPs did not include a sufficient washout period or follow-up duration for outcome assessment.

Conclusions: When using AI-generated SAPs for target trial emulation in claims databases, the most critical pitfalls identified were the failure to define the index date, improper handling of temporal-causal relationships, and a lack of NHIRD-related knowledge—despite the existence of over 5,000 publications based on this database. Future efforts should focus on pre-training AI tools in causal inference, TTE methodology, and database-specific characteristics to improve the quality of AI-generated SAPs.