mlsanityData to 14 Oct 2026

R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning

arXiv 2506.04185 ↗First posted 4 Jun 2025cs.CLcited by 34

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large language models (LLMs) have notably progressed in multi-step and long-chain reasoning. However, extending their reasoning capabilities to encompass deep interactions with search remains a non-trivial challenge, as models often fail to identify optimal reasoning-search interaction trajectories, resulting in suboptimal responses. We propose R-Search, a novel reinforcement learning framework for Reasoning-Search integration, designed to enable LLMs to autonomously execute multi-step reasoning with deep search interaction, and learn optimal reasoning search interaction trajectories via multi-reward signals, improving response quality in complex logic- and knowledge-intensive tasks. R-Search guides the LLM to dynamically decide when to retrieve or reason, while globally integrating key evidence to enhance deep knowledge interaction between reasoning and search. During RL training, R-Search provides multi-stage, multi-type rewards to jointly optimize the reasoning-search trajectory. Experiments on seven datasets show that R-Search outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain). The code and data are available at https://github.com/QingFei1/R-Search.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 31 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

28 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning12 Mar
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  3. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  9. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  10. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  12. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
13 more
  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  2. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  3. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 451
  4. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  5. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
  6. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  7. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 166
  8. 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 147
  9. 2021End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answeringcited by 133
  10. 2024Rule Based Rewards for Language Model Safetycited by 108
  11. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 78
  12. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  13. 2024LongRAG: A Dual-Perspective Retrieval-Augmented Generation Paradigm for Long-Context Question Answeringcited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 21.

34 papers.

Sort:

By area:

  1. 2025Deep Research: A Survey of Autonomous Research Agentscited by 66
  2. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 62
  3. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 34
  4. 2025ParallelSearch: Train your LLMs to Decompose Query and Search Sub-queries in Parallel with Reinforcement Learningcited by 26
  5. 2026TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMscited by 24
  6. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 22
  7. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agentscited by 16
  8. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  9. 2026SmartSearch: Process Reward-Guided Query Refinement for Search Agentscited by 14
  10. 2026RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewardscited by 12
  11. 2026Dynamic Skill Lifecycle Management for Agentic Reinforcement Learningcited by 12
  12. 2025EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notescited by 8
22 more
  1. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  2. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  3. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 7
  4. 2026OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Searchcited by 7
  5. 2025Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMscited by 6
  6. 2025Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generationcited by 6
  7. 2025Interact-RAG: Reason and Interact with the Corpus, Beyond Black-Box Retrievalcited by 6
  8. 2025From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMscited by 5
  9. 2026AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAGcited by 5
  10. 2026ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agentscited by 4
  11. 2026MultiCube-RAG for Multi-hop Question Answeringcited by 3
  12. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 3
  13. 2026SearchMaster: Grounded and Regulated Self-Play for Search Agentscited by 3
  14. 2026CAFE: Self-Improving Search Agents Need Co-Evolving Feedbackcited by 2
  15. 2025Learning to Trust: Dynamic Utilization of Retrieval-Augmented Generation for E-commerce Search Relevancecited by 0
  16. 2026SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrievalcited by 0
  17. 2026LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAGcited by 0
  18. 2026C-MIG: Multi-view Information Gain-based Retrieval-Augmented Generation for Clinical Diagnosis Reasoningcited by 0
  19. 2026CheckRLM: Effective Knowledge-Thought Coherence Checking in Retrieval-Augmented Reasoningcited by 0
  20. 2026HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learningcited by 0
  21. 2026Spurious Advantage Hidden in GRPOcited by 0
  22. 2026Legal-R1: Agentic Retrieval for Evidence-Based Legal Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
  2. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents
  3. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
  4. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  6. 2025Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models
  7. 2025RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning
  8. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle