# Can large language models replace humans in the systematic review process? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages

arXiv: https://arxiv.org/abs/2310.17526 · HTML: https://mlsanity.com/paper/2310.17526/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 41 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Human heuristics for AI-generated language are flawed](https://mlsanity.com/paper/2206.07271/) (2022)
- [A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literature](https://mlsanity.com/paper/1806.04185/) (2018)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [Automated Paper Screening for Clinical Reviews Using Large Language Models](https://mlsanity.com/paper/2305.00844/) (2023)
- [Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews](https://mlsanity.com/paper/2307.06464/) (2023)

## Built on by (most cited first, 30 of 36)

- [The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs](https://mlsanity.com/paper/2501.10970/) (2025)
- [Large Language Models in Mental Health Care: a Scoping Review](https://mlsanity.com/paper/2401.02984/) (2024)
- [Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning](https://mlsanity.com/paper/2404.08680/) (2024)
- [The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review](https://mlsanity.com/paper/2409.04600/) (2024)
- [When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings](https://mlsanity.com/paper/2407.04503/) (2024)
- [The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews](https://mlsanity.com/paper/2404.15667/) (2024)
- [A foundation model for human-AI collaboration in medical literature mining](https://mlsanity.com/paper/2501.16255/) (2025)
- [Automatically Extracting Numerical Results from Randomized Controlled Trials with Large Language Models](https://mlsanity.com/paper/2405.01686/) (2024)
- [Transforming Evidence Synthesis: A Systematic Review of the Evolution of Automated Meta-Analysis in the Age of AI](https://mlsanity.com/paper/2504.20113/) (2025)
- [Language Models as Critical Thinking Tools: A Case Study of Philosophers](https://mlsanity.com/paper/2404.04516/) (2024)
- [Labeling Free-text Data using Language Model Ensembles](https://mlsanity.com/paper/2501.08413/) (2025)
- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [Streamlining Systematic Reviews: A Novel Application of Large Language Models](https://mlsanity.com/paper/2412.15247/) (2024)
- [SafeReview: Defending LLM-based Review Systems Against Adversarial Hidden Prompts](https://mlsanity.com/paper/2604.26506/) (2026)
- [Query-driven Document-level Scientific Evidence Extraction from Biomedical Studies](https://mlsanity.com/paper/2505.06186/) (2025)
- [Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research](https://mlsanity.com/paper/2506.12691/) (2025)
- [Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification](https://mlsanity.com/paper/2510.16091/) (2025)
- [Performance of diverse evaluation metrics in NLP-based assessment and text generation of consumer complaints](https://mlsanity.com/paper/2506.21623/) (2025)
- [Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis](https://mlsanity.com/paper/2509.00038/) (2025)
- [Large language models streamline automated systematic review: A preliminary study](https://mlsanity.com/paper/2502.15702/) (2025)
- [AutoForest: Automatically Generating Forest Plots from Biomedical Studies with End-to-End Evidence Extraction and Synthesis](https://mlsanity.com/paper/2606.02403/) (2026)
- [The Lyme Disease Controversy: An AI-Driven Discourse Analysis of a Quarter Century of Academic Debate and Divides](https://mlsanity.com/paper/2504.08777/) (2025)
- [Expediting data extraction using a large language model (LLM) and scoping review protocol: a methodological study within a complex scoping review](https://mlsanity.com/paper/2507.06623/) (2025)
- [AutoEmpirical: LLM-Based Automated Research for Empirical Software Fault Analysis](https://mlsanity.com/paper/2510.04997/) (2025)
- [Strategic Innovation Management in the Age of Large Language Models Market Intelligence, Adaptive R&D, and Ethical Governance](https://mlsanity.com/paper/2511.14709/) (2025)
- [General-purpose AI models can generate actionable knowledge on agroecological crop protection](https://mlsanity.com/paper/2512.11474/) (2025)
- [Reservoir Computing inspired Matrix Multiplication-free Language Model](https://mlsanity.com/paper/2512.23145/) (2025)
- [TiAb Review Plugin: A Browser-Based Tool for AI-Assisted Study Selection in Systematic Reviews](https://mlsanity.com/paper/2604.08602/) (2026)
- [Taking a Pulse on How Generative AI is Reshaping the Software Engineering Research Landscape](https://mlsanity.com/paper/2604.11184/) (2026)
- [EviSearch: Trustworthy Extraction and Synthesis of Clinical Trial Evidence with Agents that Improve with Use](https://mlsanity.com/paper/2604.14165/) (2026)

## Abstract (arXiv)

Systematic reviews are vital for guiding practice, research, and policy, yet they are often slow and labour-intensive. Large language models (LLMs) could offer a way to speed up and automate systematic reviews, but their performance in such tasks has not been comprehensively evaluated against humans, and no study has tested GPT-4, the biggest LLM so far. This pre-registered study evaluates GPT-4's capability in title/abstract screening, full-text review, and data extraction across various literature types and languages using a 'human-out-of-the-loop' approach. Although GPT-4 had accuracy on par with human performance in most tasks, results were skewed by chance agreement and dataset imbalance. After adjusting for these, there was a moderate level of performance for data extraction, and - barring studies that used highly reliable prompts - screening performance levelled at none to moderate for different stages and languages. When screening full-text literature using highly reliable prompts, GPT-4's performance was 'almost perfect.' Penalising GPT-4 for missing key studies using highly reliable prompts improved its performance even more. Our findings indicate that, currently, substantial caution should be used if LLMs are being used to conduct systematic reviews, but suggest that, for certain systematic review tasks delivered under reliable prompts, LLMs can rival human performance.