# Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning

arXiv: https://arxiv.org/abs/2601.13115 · HTML: https://mlsanity.com/paper/2601.13115/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 76 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 27 of 27)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2503.05592/) (2025)
- [DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments](https://mlsanity.com/paper/2504.03160/) (2025)
- [Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models](https://mlsanity.com/paper/2410.07176/) (2024)
- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [Entropy-based Exploration Conduction for Multi-step Reasoning](https://mlsanity.com/paper/2503.15848/) (2025)
- [Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond](https://mlsanity.com/paper/2304.13874/) (2023)
- [Language Models that Think, Chat Better](https://mlsanity.com/paper/2509.20357/) (2025)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [IterCQR: Iterative Conversational Query Reformulation with Retrieval Guidance](https://mlsanity.com/paper/2311.09820/) (2023)
- [Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging](https://mlsanity.com/paper/2505.09316/) (2025)
- [Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning](https://mlsanity.com/paper/2510.20150/) (2025)
- [OpenDecoder: Open Large Language Model Decoding to Incorporate Document Quality in RAG](https://mlsanity.com/paper/2601.09028/) (2026)
- [ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2505.15776/) (2025)
- [REFRAG: Rethinking RAG based Decoding](https://mlsanity.com/paper/2509.01092/) (2025)
- [Blind Spot Navigation in Large Language Model Reasoning with Thought Space Explorer](https://mlsanity.com/paper/2410.24155/) (2024)
- [Parametric Retrieval-Augmented Generation using Latent Routing of LoRA Adapters](https://mlsanity.com/paper/2511.17044/) (2025)
- [ConvMix: A Mixed-Criteria Data Augmentation Framework for Conversational Dense Retrieval](https://mlsanity.com/paper/2508.04001/) (2025)
- [CRMWeaver: Building Powerful Business Agent via Agentic RL and Shared Memories](https://mlsanity.com/paper/2510.25333/) (2025)
- [ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering](https://mlsanity.com/paper/2510.13312/) (2025)

## Built on by (most cited first, 1 of 1)

- [CIR at iKAT SCAI 2026: Exploring Clarification Need Prediction in Agentic Conversational Search](https://mlsanity.com/paper/2607.19801/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have become a popular interface for human-AI interaction, supporting information seeking and task assistance through natural, multi-turn dialogue. To respond to users within multi-turn dialogues, the context-dependent user intent evolves across interactions, requiring contextual interpretation, query reformulation, and dynamic coordination between retrieval and generation. Existing studies usually follow static rewrite, retrieve, and generate pipelines, which optimize different procedures separately and overlook the mixed-initiative action optimization simultaneously. Although the recent developments in deep search agents demonstrate the effectiveness in jointly optimizing retrieval and generation via reasoning, these approaches focus on single-turn scenarios, which might lack the ability to handle multi-turn interactions. We introduce a conversational agent that interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training with tailored rewards towards evolving user goals. The experimental results across four widely used conversational benchmarks demonstrate the effectiveness of our methods by surpassing several existing strong baselines.