mlsanityData to 14 Oct 2026

An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents

arXiv 2505.15117 ↗First posted 21 May 2025cs.CL cs.AI cs.IRcited by 46

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has demonstrated strong potential in training large language models (LLMs) capable of complex reasoning for real-world problem solving. More recently, RL has been leveraged to create sophisticated LLM-based search agents that adeptly combine reasoning with search engine use. While the use of RL for training search agents is promising, the optimal design of such agents remains not fully understood. In particular, key factors -- such as (1) reward formulation, (2) the choice and characteristics of the underlying LLM, and (3) the role of the search engine in the RL process -- require further investigation. In this work, we conduct comprehensive empirical studies to systematically investigate these and offer actionable insights. We highlight several key findings: format rewards are effective in improving final performance, whereas intermediate retrieval rewards have limited impact; the scale and initialization of the LLM (general-purpose vs. reasoning-specialized) significantly influence RL outcomes; and the choice of search engine plays a critical role in shaping RL training dynamics and the robustness of the trained agent during inference. These establish important guidelines for successfully building and deploying LLM-based search agents in real-world applications. Code is available at https://github.com/PeterGriffinJin/Search-R1.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 58 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

38 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning25 Mar
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning12 Mar
  3. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning7 Mar
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  12. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
21 more
  1. 2019ELI5: Long Form Question Answeringcited by 575
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  4. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  5. 2024Measuring short-form factuality in large language modelscited by 373
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  7. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  8. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  9. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  10. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  11. 2025Agentic Retrieval-Augmented Generation: A Survey on Agentic RAGcited by 249
  12. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
  13. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  14. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  15. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasonerscited by 140
  16. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  17. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  18. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 83
  19. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
  20. 2025Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learningcited by 33
  21. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspectivecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 23.

46 papers.

Sort:

By area:

  1. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 200
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  4. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 72
  5. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  6. 2026TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMscited by 24
  7. 2026Agentic Aggregation for Parallel Scaling of Long-Horizon Agentic Taskscited by 18
  8. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  9. 2025SSRL: Self-Search Reinforcement Learningcited by 14
  10. 2025Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Searchcited by 12
  11. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 11
  12. 2025InfoFlow: Reinforcing Search Agent Via Reward Density Optimizationcited by 11
34 more
  1. 2025AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Playcited by 10
  2. 2025HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generationcited by 10
  3. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 9
  4. 2025ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoningcited by 9
  5. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimizationcited by 9
  6. 2026Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipecited by 8
  7. 2025TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Frameworkcited by 7
  8. 2025Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMscited by 6
  9. 2025Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generationcited by 6
  10. 2025MetaAgent: Toward Self-Evolving Agent via Tool Meta-Learningcited by 5
  11. 2025ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answeringcited by 4
  12. 2026AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoningcited by 4
  13. 2025RECON: Reasoning with Condensation for Efficient Retrieval-Augmented Generationcited by 3
  14. 2026S3-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Datacited by 3
  15. 2026PiCA: Pivot-Based Credit Assignment for Search Agentic Reinforcement Learningcited by 3
  16. 2026Effective Reinforcement Learning for Agentic Search by Recycling Zero-Variance Queries During Trainingcited by 3
  17. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 3
  18. 2026TCPO: Turn-Level Credit Policy Optimizationcited by 3
  19. 2025A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesiscited by 2
  20. 2025REX-RAG: Reasoning Exploration with Policy Correction in Retrieval-Augmented Generationcited by 2
  21. 2026Small Model as Master Orchestrator: Learning Unified Agent-Tool Orchestration with Parallel Subtask Decompositioncited by 2
  22. 2026Divide and Cooperate: Role-Decomposed Multi-Agent LLM Training with Cross-Agent Learning Signalscited by 2
  23. 2026Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agentscited by 2
  24. 2025Omne-R1: Learning to Reason with Memory for Multi-hop Question Answeringcited by 1
  25. 2026D2Plan: Dual-Agent Dynamic Global Planning for Complex Retrieval-Augmented Reasoningcited by 1
  26. 2026Can David Beat Goliath? On Multi-Hop Reasoning with Resource-Constrained Agentscited by 1
  27. 2025DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoningcited by 0
  28. 2026Verbal-R3: Verbal Reranker as the Missing Bridge between Retrieval and Reasoningcited by 0
  29. 2026LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAGcited by 0
  30. 2026PhantomEnvironments: Training LLM Agents in Fictional Worldscited by 0
  31. 2026Beyond Instruction Following: Learning Grounded Skill-Following with Skill Contractscited by 0
  32. 2026Programmatic Search Agents: Extending Agentic Search Beyond Query Reformulationcited by 0
  33. 2026Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignmentcited by 0
  34. 2026Beyond Outcome Rewards: Constructing and Assigning Retrieval Credit for Search Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
  2. 2026PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent
  3. 2025Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models
  4. 2025AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning
  5. 2025Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning
  6. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applications
  7. 2025Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents
  8. 2025SEM: Reinforcement Learning for Search-Efficient Large Language Models