# CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning

arXiv: https://arxiv.org/abs/2112.08558 · HTML: https://mlsanity.com/paper/2112.08558/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 30 of 31)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Large Dual Encoders Are Generalizable Retrievers](https://mlsanity.com/paper/2112.07899/) (2021)
- [QuAC : Question Answering in Context](https://mlsanity.com/paper/1808.07036/) (2018)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Open-Domain Question Answering Goes Conversational via Question Rewriting](https://mlsanity.com/paper/2010.04898/) (2020)
- [Decontextualization: Making Sentences Stand-Alone](https://mlsanity.com/paper/2102.05169/) (2021)
- [Open-Retrieval Conversational Question Answering](https://mlsanity.com/paper/2005.11364/) (2020)
- [Question Rewriting for Conversational Question Answering](https://mlsanity.com/paper/2004.14652/) (2020)
- [Few-Shot Generative Conversational Query Rewriting](https://mlsanity.com/paper/2006.05009/) (2020)
- [Query Resolution for Conversational Search with Limited Supervision](https://mlsanity.com/paper/2005.11723/) (2020)
- [Deep Communicating Agents for Abstractive Summarization](https://mlsanity.com/paper/1803.10357/) (2018)
- [Conversational Question Reformulation via Sequence-to-Sequence Architectures and Pretrained Language Models](https://mlsanity.com/paper/2004.01909/) (2020)
- [Ask the Right Questions: Active Question Reformulation with Reinforcement Learning](https://mlsanity.com/paper/1705.07830/) (2017)
- [Task-Oriented Query Reformulation with Reinforcement Learning](https://mlsanity.com/paper/1704.04572/) (2017)
- [Contextualized Query Embeddings for Conversational Search](https://mlsanity.com/paper/2104.08707/) (2021)
- [Boosting Search Engines with Interactive Agents](https://mlsanity.com/paper/2109.00527/) (2021)
- [Context Attentive Document Ranking and Query Suggestion](https://mlsanity.com/paper/1906.02329/) (2019)
- [Attention-based Hierarchical Neural Query Suggestion](https://mlsanity.com/paper/1805.02816/) (2018)

## Built on by (most cited first, 30 of 68)

- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [ChatQA: Surpassing GPT-4 on Conversational QA and RAG](https://mlsanity.com/paper/2401.10225/) (2024)
- [Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting](https://mlsanity.com/paper/2310.09716/) (2023)
- [Bridging the Preference Gap between Retrievers and LLMs](https://mlsanity.com/paper/2401.06954/) (2024)
- [ConvGQR: Generative Query Reformulation for Conversational Search](https://mlsanity.com/paper/2305.15645/) (2023)
- [Dialog Inpainting: Turning Documents into Dialogs](https://mlsanity.com/paper/2205.09073/) (2022)
- [RaFe: Ranking Feedback Improves Query Rewriting for RAG](https://mlsanity.com/paper/2405.14431/) (2024)
- [Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement](https://mlsanity.com/paper/2305.14497/) (2023)
- [CHIQ: Contextual History Enhancement for Improving Query Rewriting in Conversational Search](https://mlsanity.com/paper/2406.05013/) (2024)
- [Learning to Relate to Previous Turns in Conversational Search](https://mlsanity.com/paper/2306.02553/) (2023)
- [Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversation](https://mlsanity.com/paper/2402.11827/) (2024)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [Saving Dense Retriever from Shortcut Dependency in Conversational Search](https://mlsanity.com/paper/2202.07280/) (2022)
- [Reinforced Question Rewriting for Conversational Question Answering](https://mlsanity.com/paper/2210.15777/) (2022)
- [ChatRetriever: Adapting Large Language Models for Generalized and Robust Conversational Dense Retrieval](https://mlsanity.com/paper/2404.13556/) (2024)
- [Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action](https://mlsanity.com/paper/2405.17822/) (2024)
- [Generalizing Conversational Dense Retrieval via LLM-Cognition Data Augmentation](https://mlsanity.com/paper/2402.07092/) (2024)
- [Query Performance Prediction: From Ad-hoc to Conversational Search](https://mlsanity.com/paper/2305.10923/) (2023)
- [AdaCQR: Enhancing Query Reformulation for Conversational Search via Sparse and Dense Retrieval Alignment](https://mlsanity.com/paper/2407.01965/) (2024)
- [UniConv: Unifying Retrieval and Response Generation for Large Language Models in Conversations](https://mlsanity.com/paper/2507.07030/) (2025)
- [How to Leverage Personal Textual Knowledge for Personalized Conversational Information Retrieval](https://mlsanity.com/paper/2407.16192/) (2024)
- [Aligning Query Representation with Rewritten Query and Relevance Judgments in Conversational Search](https://mlsanity.com/paper/2407.20189/) (2024)
- [Q-TOD: A Query-driven Task-oriented Dialogue System](https://mlsanity.com/paper/2210.07564/) (2022)
- [Wizard of Shopping: Target-Oriented E-commerce Dialogue Generation with Decision Tree Branching](https://mlsanity.com/paper/2502.00969/) (2025)
- [Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI](https://mlsanity.com/paper/2308.07213/) (2023)
- [Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QA](https://mlsanity.com/paper/2409.15515/) (2024)
- [No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Users](https://mlsanity.com/paper/2410.07589/) (2024)
- [DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Search](https://mlsanity.com/paper/2410.14609/) (2024)
- [ConvSDG: Session Data Generation for Conversational Search](https://mlsanity.com/paper/2403.11335/) (2024)
- [CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation](https://mlsanity.com/paper/2410.23090/) (2024)

## Abstract (arXiv)

Compared to standard retrieval tasks, passage retrieval for conversational question answering (CQA) poses new challenges in understanding the current user question, as each question needs to be interpreted within the dialogue context. Moreover, it can be expensive to re-train well-established retrievers such as search engines that are originally developed for non-conversational queries. To facilitate their use, we develop a query rewriting model CONQRR that rewrites a conversational question in the context into a standalone question. It is trained with a novel reward function to directly optimize towards retrieval using reinforcement learning and can be adapted to any off-the-shelf retriever. CONQRR achieves state-of-the-art results on a recent open-domain CQA dataset containing conversations from three different sources, and is effective for two different off-the-shelf retrievers. Our extensive analysis also shows the robustness of CONQRR to out-of-domain dialogues as well as to zero query rewriting supervision.