# Streamlining Systematic Reviews: A Novel Application of Large Language Models

arXiv: https://arxiv.org/abs/2412.15247 · HTML: https://mlsanity.com/paper/2412.15247/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 14 Dec 2024.

## Builds on (most cited first, 5 of 5)

- [Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references](https://mlsanity.com/paper/1402.4578/) (2014)
- [Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages](https://mlsanity.com/paper/2310.17526/) (2023)
- [Open Source Software for Efficient and Transparent Reviews](https://mlsanity.com/paper/2006.12166/) (2020)
- [Automated Paper Screening for Clinical Reviews Using Large Language Models](https://mlsanity.com/paper/2305.00844/) (2023)
- [The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews](https://mlsanity.com/paper/2404.15667/) (2024)

## Built on by (most cited first, 4 of 4)

- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis](https://mlsanity.com/paper/2509.00038/) (2025)
- [Chained Prompting for Better Systematic Review Search Strategies](https://mlsanity.com/paper/2602.00011/) (2025)
- [Scientific Knowledge Discovery in the Age of Large Language Models](https://mlsanity.com/paper/2607.26670/) (2026)

## Abstract (arXiv)

Systematic reviews (SRs) are essential for evidence-based guidelines but are often limited by the time-consuming nature of literature screening. We propose and evaluate an in-house system based on Large Language Models (LLMs) for automating both title/abstract and full-text screening, addressing a critical gap in the literature. Using a completed SR on Vitamin D and falls (14,439 articles), the LLM-based system employed prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening. The system achieved an article exclusion rate (AER) of 99.5%, specificity of 99.6%, a false negative rate (FNR) of 0%, and a negative predictive value (NPV) of 100%. After screening, only 78 articles required manual review, including all 20 identified by traditional methods, reducing manual screening time by 95.5%. For comparison, Rayyan, a commercial tool for title/abstract screening, achieved an AER of 72.1% and FNR of 5% when including articles Rayyan considered as undecided or likely to include. Lowering Rayyan's inclusion thresholds improved FNR to 0% but increased screening time. By addressing both screening phases, the LLM-based system significantly outperformed Rayyan and traditional methods, reducing total screening time to 25.5 hours while maintaining high accuracy. These findings highlight the transformative potential of LLMs in SR workflows by offering a scalable, efficient, and accurate solution, particularly for the full-text screening phase, which has lacked automation tools.