3D-02 - A Comparison of Traditional versus Semi-Automated Processes for Screening and Data Extraction in a Systematic Literature Review.

Background: Traditional systematic literature reviews (SLR) involve extensive searching, screening, and laborious data extraction processes. Machine learning (ML) and natural language processing (NLP) techniques can automate and expedite SLR tasks. This study compares the consistency between SLR outputs and efficiency of manual and semi-automated processes.

Objectives: To determine the concordance rate and time spent to manually review and extract data versus automating the process in an SLR.

Methods: Traditional and semi-automated SLR approaches were employed to answer the same research question with same study parameters (population of interest, interventions, comparators, outcomes, time and settings). Literature searches were performed on relevant databases in both approaches. Two reviewers removed duplicate citations using a reference management system, manually screened title and abstracts with discrepancies resolved by consensus or a third reviewer and manually extracted relevant data. Same review was conducted using a duplication detection model, trained prediction models for title & abstract screening and full text screening, and NLP-based data extraction model developed by CapeStart. Two reviewers validated the predictions provided by the screening models and discrepancies were resolved by a third reviewer as needed. Concordance rate was calculated by dividing the total number of relevant publications in concordance between the two approaches by the number of relevant publications, then multiplying by 100. Time taken to complete the tasks in the two approaches was documented.

Results: In the manual approach, after de-duping 2507 citations, 1897 unique citations were manually screened for relevant titles and abstracts to yield 628 relevant publications. Manual full text screening resulted in 42 studies for data extraction.
In the automated process, after de-duping 2662 articles, 2871 unique citations were screened by applying a duplicate detection model. Prediction models identified relevant citations to yield 499 articles at title and abstract screening stage and 42 studies at the full text screening for data extraction. Although both approaches resulted in 42 relevant publications, the concordance rate for citations obtained from title and abstract screening was 83% which increased by 7% after full text screening. Total time spent on the task using the semi-automated approach was 14 days compared to 3 months for the manual approach.

Conclusions: A semi-automated approach, combining both manual and automated methods, provides good resource balance, leveraging the strengths of both to enhance the overall quality and efficiency of SLRs.