# The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews

arXiv: https://arxiv.org/abs/2404.15667 · HTML: https://mlsanity.com/paper/2404.15667/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages](https://mlsanity.com/paper/2310.17526/) (2023)
- [Automated Paper Screening for Clinical Reviews Using Large Language Models](https://mlsanity.com/paper/2305.00844/) (2023)
- [Successful Combination of Database Search and Snowballing for Identification of Primary Studies in Systematic Literature Studies](https://mlsanity.com/paper/2307.02612/) (2023)
- [Zero-shot Generative Large Language Models for Systematic Review Screening Automation](https://mlsanity.com/paper/2401.06320/) (2024)
- [Time Pressure in Software Engineering: A Systematic Review](https://mlsanity.com/paper/1901.05771/) (2019)
- [Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation](https://mlsanity.com/paper/2308.06610/) (2023)
- [Paving the Way for Mature Secondary Research: The Seven Types of Literature Review](https://mlsanity.com/paper/2205.01163/) (2022)
- [Automated title and abstract screening for scoping reviews using the GPT-4 Large Language Model](https://mlsanity.com/paper/2311.07918/) (2023)

## Built on by (most cited first, 14 of 14)

- [The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review](https://mlsanity.com/paper/2409.04600/) (2024)
- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [Streamlining Systematic Reviews: A Novel Application of Large Language Models](https://mlsanity.com/paper/2412.15247/) (2024)
- [AISysRev -- LLM-based Tool for Title-abstract Screening](https://mlsanity.com/paper/2510.06708/) (2025)
- [SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews](https://mlsanity.com/paper/2609.05505/) (2026)
- [Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Models](https://mlsanity.com/paper/2508.04427/) (2025)
- [Measuring the State of Open Science in Transportation Using Large Language Models](https://mlsanity.com/paper/2601.14429/) (2026)
- [On the Use of a Large Language Model to Support the Conduction of a Systematic Mapping Study: A Brief Report from a Practitioner's View](https://mlsanity.com/paper/2602.10147/) (2026)
- [Taking a Pulse on How Generative AI is Reshaping the Software Engineering Research Landscape](https://mlsanity.com/paper/2604.11184/) (2026)
- [A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions](https://mlsanity.com/paper/2605.25536/) (2026)
- [Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations](https://mlsanity.com/paper/2606.17588/) (2026)
- [Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews](https://mlsanity.com/paper/2608.14737/) (2026)
- [Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency](https://mlsanity.com/paper/2608.26885/) (2026)
- [Checkpoints Are Not Enough: Trust Calibration in CoSLR, a Human-AI System for Systematic Literature Reviews](https://mlsanity.com/paper/2609.22248/) (2026)

## Abstract (arXiv)

Systematic review (SR) is a popular research method in software engineering (SE). However, conducting an SR takes an average of 67 weeks. Thus, automating any step of the SR process could reduce the effort associated with SRs. Our objective is to investigate if Large Language Models (LLMs) can accelerate title-abstract screening by simplifying abstracts for human screeners, and automating title-abstract screening. We performed an experiment where humans screened titles and abstracts for 20 papers with both original and simplified abstracts from a prior SR. The experiment with human screeners was reproduced with GPT-3.5 and GPT-4 LLMs to perform the same screening tasks. We also studied if different prompting techniques (Zero-shot (ZS), One-shot (OS), Few-shot (FS), and Few-shot with Chain-of-Thought (FS-CoT)) improve the screening performance of LLMs. Lastly, we studied if redesigning the prompt used in the LLM reproduction of screening leads to improved performance. Text simplification did not increase the screeners' screening performance, but reduced the time used in screening. Screeners' scientific literacy skills and researcher status predict screening performance. Some LLM and prompt combinations perform as well as human screeners in the screening tasks. Our results indicate that the GPT-4 LLM is better than its predecessor, GPT-3.5. Additionally, Few-shot and One-shot prompting outperforms Zero-shot prompting. Using LLMs for text simplification in the screening process does not significantly improve human performance. Using LLMs to automate title-abstract screening seems promising, but current LLMs are not significantly more accurate than human screeners. To recommend the use of LLMs in the screening process of SRs, more research is needed. We recommend future SR studies publish replication packages with screening data to enable more conclusive experimenting with LLM screening.