# Reassessing Large Language Model Boolean Query Generation for Systematic Reviews

arXiv: https://arxiv.org/abs/2505.07155 · HTML: https://mlsanity.com/paper/2505.07155/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 26 references that are in the map.
- First posted on arXiv 12 May 2025.

## Builds on (most cited first, 8 of 8)

- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models](https://mlsanity.com/paper/2408.02442/) (2024)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [A Reproducibility and Generalizability Study of Large Language Models for Query Generation](https://mlsanity.com/paper/2411.14914/) (2024)
- [Seed-driven Document Ranking for Systematic Reviews: A Reproducibility Study](https://mlsanity.com/paper/2112.04090/) (2021)
- [Automated MeSH Term Suggestion for Effective Query Formulation in Systematic Reviews Literature Search](https://mlsanity.com/paper/2209.08687/) (2022)
- [Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation](https://mlsanity.com/paper/2309.05238/) (2023)

## Built on by (most cited first, 4 of 4)

- [Quantifying the Impact of Structured Output Format on Large Language Models through Causal Inference](https://mlsanity.com/paper/2509.21791/) (2025)
- [Search, Inspect, Fetch: Exploiting Structure-Aware Boolean Retrieval for Deep-Search Agents](https://mlsanity.com/paper/2608.02751/) (2026)
- [How Much Structure Do LLMs Need? Evaluating LLMs for Bibliometric Cluster Description](https://mlsanity.com/paper/2605.24351/) (2026)
- [EMBL AI Librarian: Life-Sciences Knowledge Layer for AI Agents](https://mlsanity.com/paper/2607.28229/) (2026)

## Abstract (arXiv)

Systematic reviews are comprehensive literature reviews that address highly focused research questions and represent the highest form of evidence in medicine. A critical step in this process is the development of complex Boolean queries to retrieve relevant literature. Given the difficulty of manually constructing these queries, recent efforts have explored Large Language Models (LLMs) to assist in their formulation. One of the first studies,Wang et al., investigated ChatGPT for this task, followed by Staudinger et al., which evaluated multiple LLMs in a reproducibility study. However, the latter overlooked several key aspects of the original work, including (i) validation of generated queries, (ii) output formatting constraints, and (iii) selection of examples for chain-of-thought (Guided) prompting. As a result, its findings diverged significantly from the original study. In this work, we systematically reproduce both studies while addressing these overlooked factors. Our results show that query effectiveness varies significantly across models and prompt designs, with guided query formulation benefiting from well-chosen seed studies. Overall, prompt design and model selection are key drivers of successful query formulation. Our findings provide a clearer understanding of LLMs' potential in Boolean query generation and highlight the importance of model- and prompt-specific optimisations. The complex nature of systematic reviews adds to challenges in both developing and reproducing methods but also highlights the importance of reproducibility studies in this domain.