mlsanityData to 14 Oct 2026

How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1

arXiv 2602.19526 ↗First posted 23 Feb 2026cs.CLcited by 8

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Deep Research agents tackle knowledge-intensive tasks through multi-round retrieval and decision-oriented generation. While reinforcement learning (RL) has been shown to improve performance in this paradigm, its contributions remain underexplored. To fully understand the role of RL, we conduct a systematic study along three decoupled dimensions: prompt template, reward function, and policy optimization. Our study reveals that: 1) the Fast Thinking template yields greater stability and better performance than the Slow Thinking template used in prior work; 2) the F1-based reward underperforms the EM due to training collapse driven by answer avoidance; this can be mitigated by incorporating action-level penalties, ultimately surpassing EM; 3) REINFORCE outperforms PPO while requiring fewer search actions, whereas GRPO shows the poorest stability among policy optimization methods. Building on these insights, we then introduce Search-R1++, a strong baseline that improves the performance of Search-R1 from 0.403 to 0.442 (Qwen2.5-7B) and 0.289 to 0.331 (Qwen2.5-3B). We hope that our findings can pave the way for more principled and reliable RL training strategies in Deep Research systems.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 40 of its 44 references that are in the map. First posted on arXiv 23 Feb 2026.

Builds on

40 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025DeepResearch-Slice: Bridging the Retrieval-Utilization Gap via Explicit Text Slicing16 Dec
  2. 2025Reinforcement Learning Foundations for Deep Research Systems: A Survey8 Sep
  3. 2025Deep Research Agents: A Systematic Examination And Roadmap22 Jun

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
25 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  4. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  5. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  7. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  9. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  12. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  13. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  14. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  15. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  16. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  17. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  18. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  19. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 74
  20. 2025A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applicationscited by 50
  21. 2025Chain-of-Retrieval Augmented Generationcited by 49
  22. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  23. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learningcited by 29
  24. 2016Improving Policy Gradient by Exploring Under-appreciated Rewardscited by 14
  25. 2025Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Studycited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Deep Research of Deep Research: From Transformer to Agent, From AI to AI for Sciencecited by 1
  2. 2026MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environmentscited by 1
  3. 2026Mint-Agent: Introducing Finance-Native Agentic Foundation Modelscited by 0
  4. 2026Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Modelscited by 0
  5. 2026DualStake: Dual-Path Confidence Calibration in Deep Research Agentscited by 0
  6. 2026Reinforcement Learning with Verifiable Rewards for Small Search Agentscited by 0
  7. 2026Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfittingcited by 0
  8. 2026SOAR: Source-Aware Reinforcement Learning with Precise Credit Assignment for Search Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking the Design of Reinforcement Learning-Based Deep Research Agents
  2. 2025DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search
  3. 2026Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
  4. 2026LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent
  5. 2026PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent
  6. 2026SynPlanResearch-R1: Encouraging Tool Exploration for Deep Research with Synthetic Plans
  7. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervision
  8. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards