(SLA-070) Innovative use of Generative Artificial Intelligence Copilot to extract data for systematic literature reviews

Background: Guidelines for literature reviews recommend data extraction by two individuals, with results compared for accuracy. In case of discrepancies, a third person should be consulted. Automatic extraction using Generative AI, such as Microsoft Copilot, could significantly enhance the efficiency and accuracy of these steps.

Objectives: To assess the effectiveness of Copilot in extracting data for systematic literature reviews.

Methods: As a first step, a systematic review following PRISMA guidelines was performed. Manually extracted data was compared to data extracted by Microsoft Copilot (GPT-4) in Teams. Data for 100 studies was extracted—20 studies at a time—including supplemental information. A prompt with 18 questions and instructions was included. Copilot’s output was compared using Excel, estimating the Accuracy, Sensitivity, Specificity, Positive Predictive Value (PPV), and Negative Predictive Value (NPV).

Results: In total, the Accuracy was 98.4%, Sensitivity 98.7%, Specificity 97.5%, PPV 99.2%, and NPV 95.8%. Basic information questions (N=12) achieved 100% accuracy (e.g., author, year, N). Sixteen of the questions had a PPV and NPV higher than 90%. Two questions presented difficulties: the timing of outcome assessment (NPV = 65.7%) and a list of validated study references (PPV = 60%). If authors did not explicitly provide this information, the AI tool struggled to retrieve it accurately.

Conclusions: These results demonstrate the overall effectiveness of Copilot for data extraction and the potential of generative AI to enhance efficiency and quality in systematic reviews and meta-analyses. The challenges encountered highlight the importance of prompt refinements and tool training. Complete reliability, accuracy, and reproducibility cannot be guaranteed at this stage, necessitating human validation of all responses. At this time, generative AI can serve as an invaluable tool for quality control. A person should extract the data first, then Generative AI can be used to extract the data independently, and a second person should resolve inconsistencies. During this presentation, a live demonstration will showcase how Copilot can be utilized for data extraction and how to construct prompts to obtain accurate information.