mlsanityData to 14 Oct 2026

DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments

arXiv 2504.03160 ↗First posted 4 Apr 2025cs.AI cs.CL cs.LGcited by 286

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Large Language Models (LLMs) equipped with web search capabilities have demonstrated impressive potential for deep research tasks. However, current approaches predominantly rely on either manually engineered prompts (prompt engineering-based) with brittle performance or reinforcement learning within controlled Retrieval-Augmented Generation (RAG) environments (RAG-based) that fail to capture the complexities of real-world interaction. In this paper, we introduce DeepResearcher, the first comprehensive framework for end-to-end training of LLM-based deep research agents through scaling reinforcement learning (RL) in real-world environments with authentic web search interactions. Unlike RAG-based approaches that assume all necessary information exists within a fixed corpus, our method trains agents to navigate the noisy, unstructured, and dynamic nature of the open web. We implement a specialized multi-agent architecture where browsing agents extract relevant information from various webpage structures and overcoming significant technical challenges. Extensive experiments on open-domain research tasks demonstrate that DeepResearcher achieves substantial improvements of up to 28.9 points over prompt engineering-based baselines and up to 7.2 points over RAG-based RL agents. Our qualitative analysis reveals emergent cognitive behaviors from end-to-end RL training, including the ability to formulate plans, cross-validate information from multiple sources, engage in self-reflection to redirect research, and maintain honesty when unable to find definitive answers. Our results highlight that end-to-end training in real-world web environments is not merely an implementation detail but a fundamental requirement for developing robust research capabilities aligned with real-world applications. We release DeepResearcher at https://github.com/GAIR-NLP/DeepResearcher.

Key facts. Cited by 286 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 16 of its 36 references that are in the map. First posted on arXiv 4 Apr 2025.

Builds on

16 of 36 references.

References · 16

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  10. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  11. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 433
  12. 2025LIMR: Less is More for RL Scalingcited by 139
4 more
  1. 2025AgentRefine: Enhancing Agent Generalization through Refinement Tuningcited by 53
  2. 2025Open Deep Search: Democratizing Search with Open-source Reasoning Agentscited by 49
  3. 2024OpenResearcher: Unleashing AI for Accelerated Scientific Researchcited by 24
  4. 2024DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Searchcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 144; 2026: 142.

286 papers.

Sort:

By area:

  1. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  2. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  3. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  4. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  5. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 208
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  7. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 188
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  9. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 150
  10. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 127
  11. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  12. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
38 more
  1. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 124
  2. 2025WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agentcited by 119
  3. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  5. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  6. 2025MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scalingcited by 81
  7. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
  8. 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learningcited by 72
  9. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 70
  10. 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 69
  11. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  12. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 62
  13. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 62
  14. 2025FutureX: An Advanced Live Benchmark for LLM Agents in Future Predictioncited by 59
  15. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 57
  16. 2025SimpleDeepSearcher: Deep Information Seeking via Web-Powered Reasoning Trajectory Synthesiscited by 55
  17. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 55
  18. 2025Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learningcited by 52
  19. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agentscited by 51
  20. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  21. 2025DeepAnalyze: Agentic Large Language Models for Autonomous Data Sciencecited by 50
  22. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  23. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  24. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  25. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 46
  26. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 44
  27. 2025DeepMMSearch-R1: Empowering Multimodal LLMs in Multimodal Web Searchcited by 42
  28. 2025ReportBench: Evaluating Deep Research Agents via Academic Survey Taskscited by 38
  29. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 38
  30. 2025Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoningcited by 35
  31. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  32. 2025Mitigating Overthinking in Large Reasoning Models via Manifold Steeringcited by 33
  33. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  34. 2026MiroThinker-1.7 & H1: Towards Heavy-Duty Research Agents via Verificationcited by 32
  35. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 31
  36. 2025Characterizing Deep Research: A Benchmark and Formal Definitioncited by 30
  37. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 28
  38. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27

50 most cited of 286 All 286 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents
  2. 2025Deep Research: A Survey of Autonomous Research Agents
  3. 2025Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs
  4. 2025Deep Research: A Systematic Survey
  5. 2025Deep Research Bench: Evaluating AI Web Research Agents
  6. 2026VSearcher: Long-Horizon Multimodal Search Agent via Reinforcement Learning
  7. 2026DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent
  8. 2026Boosting Deepresearch and LongContext Ability with Self-Generated Deepresearch Rollouts Traces