# Generative Query Reformulation Using Ensemble Prompting, Document Fusion, and Relevance Feedback

arXiv: https://arxiv.org/abs/2405.17658 · HTML: https://mlsanity.com/paper/2405.17658/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 49 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 28 of 28)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [On Decoding Strategies for Neural Text Generators](https://mlsanity.com/paper/2203.15721/) (2022)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [ConvGQR: Generative Query Reformulation for Conversational Search](https://mlsanity.com/paper/2305.15645/) (2023)
- [Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedback](https://mlsanity.com/paper/2108.13454/) (2021)
- [Identifying Well-formed Natural Language Questions](https://mlsanity.com/paper/1808.09419/) (2018)
- [Simplified Data Wrangling with ir_datasets](https://mlsanity.com/paper/2103.02280/) (2021)
- [Pseudo Relevance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls](https://mlsanity.com/paper/2108.11044/) (2021)
- [When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets](https://mlsanity.com/paper/2309.08541/) (2023)
- [Generative Query Reformulation for Effective Adhoc Search](https://mlsanity.com/paper/2308.00415/) (2023)
- [GenQREnsemble: Zero-Shot LLM Ensemble Prompting for Generative Query Reformulation](https://mlsanity.com/paper/2404.03746/) (2024)
- [Diversity driven Query Rewriting in Search Advertising](https://mlsanity.com/paper/2106.03816/) (2021)
- [How to Ask Better Questions? A Large-Scale Multi-Domain Dataset for Rewriting Ill-Formed Questions](https://mlsanity.com/paper/1911.09247/) (2019)
- [An Interactive Query Generation Assistant using LLM-based Prompt Modification and User Feedback](https://mlsanity.com/paper/2311.11226/) (2023)
- [QueryExplorer: An Interactive Query Generation Assistant for Search and Exploration](https://mlsanity.com/paper/2403.15667/) (2024)

## Built on by (most cited first, 5 of 5)

- [Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs](https://mlsanity.com/paper/2503.14286/) (2025)
- [Think Then Embed: Generative Context Improves Multimodal Embedding](https://mlsanity.com/paper/2510.05014/) (2025)
- [InsertRank: LLMs can reason over BM25 scores to Improve Listwise Reranking](https://mlsanity.com/paper/2506.14086/) (2025)
- [A New Query Expansion Approach via Agent-Mediated Dialogic Inquiry](https://mlsanity.com/paper/2502.08557/) (2025)
- [PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models](https://mlsanity.com/paper/2412.05339/) (2024)

## Abstract (arXiv)

Query Reformulation (QR) is a set of techniques used to transform a user's original search query to a text that better aligns with the user's intent and improves their search experience. Recently, zero-shot QR has been a promising approach due to its ability to exploit knowledge inherent in large language models. Inspired by the success of ensemble prompting strategies which have benefited other tasks, we investigate if they can improve query reformulation. In this context, we propose two ensemble-based prompting techniques, GenQREnsemble and GenQRFusion which leverage paraphrases of a zero-shot instruction to generate multiple sets of keywords to improve retrieval performance ultimately. We further introduce their post-retrieval variants to incorporate relevance feedback from a variety of sources, including an oracle simulating a human user and a "critic" LLM. We demonstrate that an ensemble of query reformulations can improve retrieval effectiveness by up to 18% on nDCG@10 in pre-retrieval settings and 9% on post-retrieval settings on multiple benchmarks, outperforming all previously reported SOTA results. We perform subsequent analyses to investigate the effects of feedback documents, incorporate domain-specific instructions, filter reformulations, and generate fluent reformulations that might be more beneficial to human searchers. Together, the techniques and the results presented in this paper establish a new state of the art in automated query reformulation for retrieval and suggest promising directions for future research.