# QueryGym: A Toolkit for Reproducible LLM-Based Query Reformulation

arXiv: https://arxiv.org/abs/2511.15996 · HTML: https://mlsanity.com/paper/2511.15996/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 20 references that are in the map.
- First posted on arXiv 20 Nov 2025.

## Builds on (most cited first, 6 of 6)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Declarative Experimentation in Information Retrieval using PyTerrier](https://mlsanity.com/paper/2007.14271/) (2020)
- [Generative Query Reformulation for Effective Adhoc Search](https://mlsanity.com/paper/2308.00415/) (2023)
- [GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation](https://mlsanity.com/paper/2404.03746/) (2024)
- [Exploring the Best Practices of Query Expansion with Large Language Models](https://mlsanity.com/paper/2401.06311/) (2024)

## Built on by (most cited first, 5 of 5)

- [A Reproducibility Study of LLM-Based Query Reformulation](https://mlsanity.com/paper/2604.27421/) (2026)
- [A Picture of Agentic Search](https://mlsanity.com/paper/2602.17518/) (2026)
- [Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines](https://mlsanity.com/paper/2604.22661/) (2026)
- [ADORE: Iterative Query Expansion with Retrieval-Grounded Relevance Feedback](https://mlsanity.com/paper/2606.13905/) (2026)
- [A Systematic Study of Pseudo-Relevance Feedback with LLMs](https://mlsanity.com/paper/2603.11008/) (2026)

## Abstract (arXiv)

We present QueryGym, a lightweight, extensible Python toolkit that supports large language model (LLM)-based query reformulation. This is an important tool development since recent work on llm-based query reformulation has shown notable increase in retrieval effectiveness. However, while different authors have sporadically shared the implementation of their methods, there is no unified toolkit that provides a consistent implementation of such methods, which hinders fair comparison, rapid experimentation, consistent benchmarking and reliable deployment. QueryGym addresses this gap by providing a unified framework for implementing, executing, and comparing llm-based reformulation methods. The toolkit offers: (1) a Python API for applying diverse LLM-based methods, (2) a retrieval-agnostic interface supporting integration with backends such as Pyserini and PyTerrier, (3) a centralized prompt management system with versioning and metadata tracking, (4) built-in support for benchmarks like BEIR and MS MARCO, and (5) a completely open-source extensible implementation available to all researchers. QueryGym is publicly available at https://github.com/radinhamidi/QueryGym.