# High-performance automated abstract screening with large language model ensembles

arXiv: https://arxiv.org/abs/2411.02451 · HTML: https://mlsanity.com/paper/2411.02451/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2024.

## Built on by (most cited first, 8 of 8)

- [A foundation model for human-AI collaboration in medical literature mining](https://mlsanity.com/paper/2501.16255/) (2025)
- [Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification](https://mlsanity.com/paper/2510.16091/) (2025)
- [SciLitBench: Benchmark and Design Principles for LLM-Powered Systematic Literature Reviews](https://mlsanity.com/paper/2609.05505/) (2026)
- [ImpReSS: Implicit Recommender System for Support Conversations](https://mlsanity.com/paper/2506.14231/) (2025)
- [ZPD-SCA: Unveiling the Blind Spots of LLMs in Assessing Students' Cognitive Abilities](https://mlsanity.com/paper/2508.14377/) (2025)
- [Leveraging LLMs for Title and Abstract Screening for Systematic Review: A Cost-Effective Dynamic Few-Shot Learning Approach](https://mlsanity.com/paper/2512.11261/) (2025)
- [An ambiguity taxonomy for evaluating large language model performance on clinical registry abstraction: a multi-site prospective study](https://mlsanity.com/paper/2608.20373/) (2026)
- [Evaluating human and LLM screening workflows in a conceptually complex scoping review: Recall--workload trade-offs and run-to-run consistency](https://mlsanity.com/paper/2608.26885/) (2026)

## Abstract (arXiv)

Large language models (LLMs) excel in tasks requiring processing and interpretation of input text. Abstract screening is a labour-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies identified by a literature search. Here, LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialled on systematic reviews in a full issue of the Cochrane Library to evaluate their accuracy in zero-shot binary classification for abstract screening. Trials over a subset of 800 records identified optimal prompting strategies and demonstrated superior performance of LLMs to human researchers in terms of sensitivity (LLM-max = 1.000, human-max = 0.775), precision (LLM-max = 0.927, human-max = 0.911), and balanced accuracy (LLM-max = 0.904, human-max = 0.865). The best performing LLM-prompt combinations were trialled across every replicated search result (n = 119,691), and exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096). 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458, with less observed performance drop in larger trials. Significant variation in performance was observed between reviews, highlighting the importance of domain-specific validation before deployment. LLMs may reduce the human labour cost of systematic review with maintained or improved accuracy and sensitivity. Systematic review is the foundation of evidence synthesis across academic disciplines, including evidence-based medicine, and LLMs may increase the efficiency and quality of this mode of research.