# ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning

arXiv: https://arxiv.org/abs/2505.15776 · HTML: https://mlsanity.com/paper/2505.15776/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 3 of its 50 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 3 of 3)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ZeroSearch: Incentivize the Search Capability of LLMs without Searching](https://mlsanity.com/paper/2505.04588/) (2025)
- [A Comparison of Question Rewriting Methods for Conversational Passage Retrieval](https://mlsanity.com/paper/2101.07382/) (2021)

## Built on by (most cited first, 9 of 9)

- [Think Then Embed: Generative Context Improves Multimodal Embedding](https://mlsanity.com/paper/2510.05014/) (2025)
- [AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Construction](https://mlsanity.com/paper/2510.15339/) (2025)
- [Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models](https://mlsanity.com/paper/2511.07581/) (2025)
- [ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering](https://mlsanity.com/paper/2510.13312/) (2025)
- [Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2601.13115/) (2026)
- [Context-Picker: Dynamic context selection using multi-stage reinforcement learning](https://mlsanity.com/paper/2512.14465/) (2025)
- [BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment](https://mlsanity.com/paper/2604.07201/) (2026)
- [CompCQR: Compositional Query Generation for Training-Free Conversational Search](https://mlsanity.com/paper/2609.14646/) (2026)
- [Rewrite What Matters: Adaptive Multilingual Query Rewriting for Reasoning via Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.04899/) (2026)

## Abstract (arXiv)

Conversational search systems require effective handling of context-dependent queries that often contain ambiguity, omission, and coreference. Conversational Query Reformulation (CQR) addresses this challenge by transforming these queries into self-contained forms suitable for off-the-shelf retrievers. However, existing CQR approaches suffer from two critical constraints: high dependency on costly external supervision from human annotations or large language models, and insufficient alignment between the rewriting model and downstream retrievers. We present ConvSearch-R1, the first self-driven framework that completely eliminates dependency on external rewrite supervision by leveraging reinforcement learning to optimize reformulation directly through retrieval signals. Our novel two-stage approach combines Self-Driven Policy Warm-Up to address the cold-start problem through retrieval-guided self-distillation, followed by Retrieval-Guided Reinforcement Learning with a specially designed rank-incentive reward shaping mechanism that addresses the sparsity issue in conventional retrieval metrics. Extensive experiments on TopiOCQA and QReCC datasets demonstrate that ConvSearch-R1 significantly outperforms previous state-of-the-art methods, achieving over 10% improvement on the challenging TopiOCQA dataset while using smaller 3B parameter models without any external supervision.