# A Reproducibility and Generalizability Study of Large Language Models for Query Generation

arXiv: https://arxiv.org/abs/2411.14914 · HTML: https://mlsanity.com/paper/2411.14914/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2024.

## Built on by (most cited first, 10 of 10)

- [Benchmarking LLM-based Relevance Judgment Methods](https://mlsanity.com/paper/2504.12558/) (2025)
- [Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies](https://mlsanity.com/paper/2510.25506/) (2025)
- [AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents](https://mlsanity.com/paper/2512.22387/) (2025)
- [Reassessing Large Language Model Boolean Query Generation for Systematic Reviews](https://mlsanity.com/paper/2505.07155/) (2025)
- [Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation](https://mlsanity.com/paper/2507.02306/) (2025)
- [Compiling Prompts, Not Crafting Them: A Reproducible Workflow for AI-Assisted Evidence Synthesis](https://mlsanity.com/paper/2509.00038/) (2025)
- [Advertising in AI systems: Society must be vigilant](https://mlsanity.com/paper/2505.18425/) (2025)
- [A Reproducible Optimisation Protocol for Calibrating Prompt-Based Large Language Model Workflows in Evidence Synthesis](https://mlsanity.com/paper/2605.06937/) (2026)
- [LLM-Generated Counterfactual Stress Scenarios for Portfolio Risk Simulation via Hybrid Prompt-RAG Pipeline](https://mlsanity.com/paper/2512.07867/) (2025)
- [AInterviewer: A Platform for Designing and Conducting AI-led Qualitative Interviews](https://mlsanity.com/paper/2606.20588/) (2026)

## Abstract (arXiv)

Systematic literature reviews (SLRs) are a cornerstone of academic research, yet they are often labour-intensive and time-consuming due to the detailed literature curation process. The advent of generative AI and large language models (LLMs) promises to revolutionize this process by assisting researchers in several tedious tasks, one of them being the generation of effective Boolean queries that will select the publications to consider including in a review. This paper presents an extensive study of Boolean query generation using LLMs for systematic reviews, reproducing and extending the work of Wang et al. and Alaniz et al. Our study investigates the replicability and reliability of results achieved using ChatGPT and compares its performance with open-source alternatives like Mistral and Zephyr to provide a more comprehensive analysis of LLMs for query generation. Therefore, we implemented a pipeline, which automatically creates a Boolean query for a given review topic by using a previously defined LLM, retrieves all documents for this query from the PubMed database and then evaluates the results. With this pipeline we first assess whether the results obtained using ChatGPT for query generation are reproducible and consistent. We then generalize our results by analyzing and evaluating open-source models and evaluating their efficacy in generating Boolean queries. Finally, we conduct a failure analysis to identify and discuss the limitations and shortcomings of using LLMs for Boolean query generation. This examination helps to understand the gaps and potential areas for improvement in the application of LLMs to information retrieval tasks. Our findings highlight the strengths, limitations, and potential of LLMs in the domain of information retrieval and literature review automation.