# A Reproducibility Study of LLM-Based Query Reformulation

arXiv: https://arxiv.org/abs/2604.27421 · HTML: https://mlsanity.com/paper/2604.27421/

## Key facts

- Cited by 2 papers in the map (data through 7 Oct 2026).
- Builds on 14 of its 37 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 14 of 14)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective](https://mlsanity.com/paper/2205.04733/) (2022)
- [ConvGQR: Generative Query Reformulation for Conversational Search](https://mlsanity.com/paper/2305.15645/) (2023)
- [Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models](https://mlsanity.com/paper/1904.09171/) (2019)
- [Generative Query Reformulation for Effective Adhoc Search](https://mlsanity.com/paper/2308.00415/) (2023)
- [GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation](https://mlsanity.com/paper/2404.03746/) (2024)
- [How Deep is your Learning: the DL-HARD Annotated Deep Learning Dataset](https://mlsanity.com/paper/2105.07975/) (2021)
- [Exploring the Best Practices of Query Expansion with Large Language Models](https://mlsanity.com/paper/2401.06311/) (2024)
- [Corpus-Steered Query Expansion with Large Language Models](https://mlsanity.com/paper/2402.18031/) (2024)
- [QueryGym: A Toolkit for Reproducible LLM-Based Query Reformulation](https://mlsanity.com/paper/2511.15996/) (2025)
- [Improving Query Expansion Using WordNet](https://mlsanity.com/paper/1309.4938/) (2013)

## Built on by (most cited first, 2 of 2)

- [ADORE: Iterative Query Expansion with Retrieval-Grounded Relevance Feedback](https://mlsanity.com/paper/2606.13905/) (2026)
- [STORM: Stepwise Token Optimization with Reward-Guided Beam Search](https://mlsanity.com/paper/2606.10621/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are now widely used for query reformulation and expansion in Information Retrieval, with many studies reporting substantial effectiveness gains. However, these results are typically obtained under heterogeneous experimental conditions, making it difficult to assess which findings are reproducible and which depend on specific implementation choices. In this work, we present a systematic reproducibility and comparative study of ten representative LLM-based query reformulation methods under a unified and strictly controlled experimental framework. We evaluate methods across two architectural LLM families at two parameter scales, three retrieval paradigms (lexical, learned sparse, and dense), and nine benchmark datasets spanning TREC Deep Learning and BEIR. Our results show that reformulation gains are strongly conditioned on the retrieval paradigm, that improvements observed under lexical retrieval do not consistently transfer to neural retrievers, and that larger LLMs do not uniformly yield better downstream performance. These findings clarify the stability and limits of reported gains in prior work. To enable transparent replication and ongoing comparison, we release all prompts, configurations, evaluation scripts, and run files through QueryGym, an open-source reformulation toolkit with a public leaderboard.footnotehttps://leaderboard.querygym.com