# Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation

arXiv: https://arxiv.org/abs/2309.05238 · HTML: https://mlsanity.com/paper/2309.05238/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 84 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [Query2doc: Query Expansion with Large Language Models](https://mlsanity.com/paper/2303.07678/) (2023)
- [Retrieval-Augmented Multimodal Language Modeling](https://mlsanity.com/paper/2211.12561/) (2022)
- [ChatGPT is not all you need. A State of the Art Review of large Generative AI models](https://mlsanity.com/paper/2301.04655/) (2023)
- [Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline](https://mlsanity.com/paper/2101.08751/) (2021)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [Understanding the Behaviors of BERT in Ranking](https://mlsanity.com/paper/1904.07531/) (2019)
- [ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots](https://mlsanity.com/paper/2302.06466/) (2023)
- [Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferences](https://mlsanity.com/paper/2303.07610/) (2023)
- [Investigating Memorization of Conspiracy Theories in Text Generation](https://mlsanity.com/paper/2101.00379/) (2021)
- [Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review](https://mlsanity.com/paper/2105.01044/) (2021)
- [From Little Things Big Things Grow: A Collection with Seed Studies for Medical Systematic Review Literature Search](https://mlsanity.com/paper/2204.03096/) (2022)
- [Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search](https://mlsanity.com/paper/2212.09017/) (2022)
- [Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study](https://mlsanity.com/paper/2112.04090/) (2021)
- [Automated MeSH Term Suggestion for Effective Query Formulation in Systematic Reviews Literature Search](https://mlsanity.com/paper/2209.08687/) (2022)

## Built on by (most cited first, 6 of 6)

- [The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review](https://mlsanity.com/paper/2409.04600/) (2024)
- [Zero-shot Generative Large Language Models for Systematic Review Screening Automation](https://mlsanity.com/paper/2401.06320/) (2024)
- [ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search](https://mlsanity.com/paper/2401.17645/) (2024)
- [Reassessing Large Language Model Boolean Query Generation for Systematic Reviews](https://mlsanity.com/paper/2505.07155/) (2025)
- [Search, Inspect, Fetch: Exploiting Structure-Aware Boolean Retrieval for Deep-Search Agents](https://mlsanity.com/paper/2608.02751/) (2026)
- [Large Language Models for Stemming: Promises, Pitfalls and Failures](https://mlsanity.com/paper/2402.11757/) (2024)

## Abstract (arXiv)

Screening prioritisation in medical systematic reviews aims to rank the set of documents retrieved by complex Boolean queries. Prioritising the most important documents ensures that subsequent review steps can be carried out more efficiently and effectively. The current state of the art uses the final title of the review as a query to rank the documents using BERT-based neural rankers. However, the final title is only formulated at the end of the review process, which makes this approach impractical as it relies on ex post facto information. At the time of screening, only a rough working title is available, with which the BERT-based ranker performs significantly worse than with the final title. In this paper, we explore alternative sources of queries for prioritising screening, such as the Boolean query used to retrieve the documents to be screened and queries generated by instruction-based generative large-scale language models such as ChatGPT and Alpaca. Our best approach is not only viable based on the information available at the time of screening, but also has similar effectiveness to the final title.