# Zero-shot Generative Large Language Models for Systematic Review Screening Automation

arXiv: https://arxiv.org/abs/2401.06320 · HTML: https://mlsanity.com/paper/2401.06320/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 65 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Builds on (most cited first, 23 of 23)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2309.14717/) (2023)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review](https://mlsanity.com/paper/2105.01044/) (2021)
- [Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM](https://mlsanity.com/paper/2304.11872/) (2023)
- [From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search](https://mlsanity.com/paper/2204.03096/) (2022)
- [Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews](https://mlsanity.com/paper/2307.06464/) (2023)
- [Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search](https://mlsanity.com/paper/2212.09017/) (2022)
- [Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation](https://mlsanity.com/paper/2308.06610/) (2023)
- [Automated MeSH Term Suggestion for Effective Query Formulation in Systematic Reviews Literature Search](https://mlsanity.com/paper/2209.08687/) (2022)
- [Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation](https://mlsanity.com/paper/2309.05238/) (2023)
- [Technology Assisted Reviews: Finding the Last Few Relevant Documents by Asking Yes/No Questions to Reviewers](https://mlsanity.com/paper/1810.05414/) (2018)
- [Neural language models for text classification in evidence-based medicine](https://mlsanity.com/paper/2012.00584/) (2020)
- [From "Identical" to "Similar": Fusing Retrieved Lists Based on Inter-Document Similarities](https://mlsanity.com/paper/1401.3883/) (2014)

## Built on by (most cited first, 14 of 14)

- [The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review](https://mlsanity.com/paper/2409.04600/) (2024)
- [FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation](https://mlsanity.com/paper/2402.11891/) (2024)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews](https://mlsanity.com/paper/2404.15667/) (2024)
- [ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search](https://mlsanity.com/paper/2401.17645/) (2024)
- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [Measuring Risk of Bias in Biomedical Reports: The RoBBR Benchmark](https://mlsanity.com/paper/2411.18831/) (2024)
- [Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System](https://mlsanity.com/paper/2505.20310/) (2025)
- [Large Language Models for Stemming: Promises, Pitfalls and Failures](https://mlsanity.com/paper/2402.11757/) (2024)
- [Investigating Large Language Models' Linguistic Abilities for Text Preprocessing](https://mlsanity.com/paper/2510.11482/) (2025)
- [Leveraging LLMs for Title and Abstract Screening for Systematic Review: A Cost-Effective Dynamic Few-Shot Learning Approach](https://mlsanity.com/paper/2512.11261/) (2025)
- [Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews](https://mlsanity.com/paper/2608.14551/) (2026)
- [Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews](https://mlsanity.com/paper/2608.14737/) (2026)
- [Checkpoints Are Not Enough: Trust Calibration in CoSLR, a Human-AI System for Systematic Literature Reviews](https://mlsanity.com/paper/2609.22248/) (2026)

## Abstract (arXiv)

Systematic reviews are crucial for evidence-based medicine as they comprehensively analyse published research findings on specific questions. Conducting such reviews is often resource- and time-intensive, especially in the screening phase, where abstracts of publications are assessed for inclusion in a review. This study investigates the effectiveness of using zero-shot large language models~(LLMs) for automatic screening. We evaluate the effectiveness of eight different LLMs and investigate a calibration technique that uses a predefined recall threshold to determine whether a publication should be included in a systematic review. Our comprehensive evaluation using five standard test collections shows that instruction fine-tuning plays an important role in screening, that calibration renders LLMs practical for achieving a targeted recall, and that combining both with an ensemble of zero-shot models saves significant screening time compared to state-of-the-art approaches.