mlsanityData to 14 Oct 2026

ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

arXiv 2503.19470 ↗First posted 25 Mar 2025cs.AI cs.CLcited by 161

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable capabilities in reasoning, exemplified by the success of OpenAI-o1 and DeepSeek-R1. However, integrating reasoning with external search processes remains challenging, especially for complex multi-hop questions requiring multiple retrieval steps. We propose ReSearch, a novel framework that trains LLMs to Reason with Search via reinforcement learning without using any supervised data on reasoning steps. Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking, and search results subsequently influence further reasoning. We train ReSearch on Qwen2.5-7B(-Instruct) and Qwen2.5-32B(-Instruct) models and conduct extensive experiments. Despite being trained on only one dataset, our models demonstrate strong generalizability across various benchmarks. Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction during the reinforcement learning process.

Key facts. Cited by 161 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 42 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

38 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning12 Mar
  2. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning7 Mar
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  4. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
22 more
  1. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  2. 2025s1: Simple test-time scalingcited by 1,341
  3. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  10. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  13. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  14. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  15. 2024Reinforcement learningcited by 247
  16. 2024Corrective Retrieval Augmented Generationcited by 197
  17. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 166
  18. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 147
  19. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  20. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 42
  21. 2024Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphscited by 24
  22. 2024Baichuan Alignment Technical Reportcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 84; 2026: 77.

161 papers.

Sort:

By area:

  1. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025Agentic Reinforced Policy Optimizationcited by 172
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  6. 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 121
  7. 2025Thinkless: LLM Learns When to Thinkcited by 113
  8. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 110
  9. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
  10. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  11. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
  12. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 79
38 more
  1. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 72
  2. 2025VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learningcited by 68
  3. 2025DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RLcited by 55
  4. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  5. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agentscited by 51
  6. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  7. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 46
  8. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  9. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 38
  10. 2025Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoningcited by 35
  11. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  12. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 33
  13. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learningcited by 33
  14. 2025MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?cited by 28
  15. 2025Synergizing RAG and Reasoning: A Systematic Reviewcited by 27
  16. 2025Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Rewardcited by 25
  17. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 23
  18. 2025DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learningcited by 23
  19. 2025GEM: A Gym for Agentic LLMscited by 23
  20. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  21. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 21
  22. 2025R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learningcited by 20
  23. 2025Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learningcited by 17
  24. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 17
  25. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  26. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  27. 2025SSRL: Self-Search Reinforcement Learningcited by 14
  28. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  29. 2025The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucinationcited by 14
  30. 2026Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoningcited by 14
  31. 2025ReaRAG: Knowledge-guided Reasoning Enhances Factuality of Large Reasoning Models with Iterative Retrieval Augmented Generationcited by 13
  32. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROIcited by 12
  33. 2025Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Searchcited by 12
  34. 2026MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chainscited by 12
  35. 2026RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewardscited by 12
  36. 2025RARE: Retrieval-Augmented Reasoning Modelingcited by 11
  37. 2025Agent Safety Alignment via Reinforcement Learningcited by 11
  38. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
  2. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models
  3. 2026Learning to Reason Efficiently with A* Post-Training
  4. 2024Navigating the Labyrinth: Evaluating LLMs' Ability to Reason About Search Problems
  5. 2025ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Context
  6. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains
  7. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuition
  8. 2025Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models