# Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation

arXiv: https://arxiv.org/abs/2308.06610 · HTML: https://mlsanity.com/paper/2308.06610/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline](https://mlsanity.com/paper/2101.08751/) (2021)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [Automated Paper Screening for Clinical Reviews Using Large Language Models](https://mlsanity.com/paper/2305.00844/) (2023)
- [Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews](https://mlsanity.com/paper/2307.06464/) (2023)
- [Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search](https://mlsanity.com/paper/2212.09017/) (2022)
- [Continuous Active Learning Using Pretrained Transformers](https://mlsanity.com/paper/2208.06955/) (2022)

## Built on by (most cited first, 7 of 7)

- [Artificial Intelligence for Literature Reviews: Opportunities and Challenges](https://mlsanity.com/paper/2402.08565/) (2024)
- [The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review](https://mlsanity.com/paper/2409.04600/) (2024)
- [Zero-shot Generative Large Language Models for Systematic Review Screening Automation](https://mlsanity.com/paper/2401.06320/) (2024)
- [The Promise and Challenges of Using LLMs to Accelerate the Screening Process of Systematic Reviews](https://mlsanity.com/paper/2404.15667/) (2024)
- [From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems](https://mlsanity.com/paper/2503.01424/) (2025)
- [LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews](https://mlsanity.com/paper/2511.12635/) (2025)
- [Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews](https://mlsanity.com/paper/2607.24991/) (2026)

## Abstract (arXiv)

Medical systematic reviews can be very costly and resource intensive. We explore how Large Language Models (LLMs) can support and be trained to perform literature screening when provided with a detailed set of selection criteria. Specifically, we instruction tune LLaMA and Guanaco models to perform abstract screening for medical systematic reviews. Our best model, Bio-SIEVE, outperforms both ChatGPT and trained traditional approaches, and generalises better across medical domains. However, there remains the challenge of adapting the model to safety-first scenarios. We also explore the impact of multi-task training with Bio-SIEVE-Multi, including tasks such as PICO extraction and exclusion reasoning, but find that it is unable to match single-task Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards specialising LLMs for the biomedical systematic review process and explore its future developmental opportunities. We release our models, code and a list of DOIs to reconstruct our dataset for reproducibility.