mlsanityData to 7 Oct 2026

R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning

arXiv 2505.17005 ↗First posted 22 May 2025cs.CL cs.AI cs.IRcited by 30

On the map: Retrieval-augmented generation → Agentic RAG, agents and memory

Abstract (arXiv)

Large Language Models (LLMs) are powerful but prone to hallucinations due to static knowledge. Retrieval-Augmented Generation (RAG) helps by injecting external information, but current methods often are costly, generalize poorly, or ignore the internal knowledge of the model. In this paper, we introduce R1-Searcher++, a novel framework designed to train LLMs to adaptively leverage both internal and external knowledge sources. R1-Searcher++ employs a two-stage training strategy: an initial SFT Cold-start phase for preliminary format learning, followed by RL for Dynamic Knowledge Acquisition. The RL stage uses outcome-supervision to encourage exploration, incorporates a reward mechanism for internal knowledge utilization, and integrates a memorization mechanism to continuously assimilate retrieved information, thereby enriching the model's internal knowledge. By leveraging internal knowledge and external search engine, the model continuously improves its capabilities, enabling efficient retrieval-augmented reasoning. Our experiments demonstrate that R1-Searcher++ outperforms previous RAG and reasoning methods and achieves efficient retrieval. The code is available at https://github.com/RUCAIBox/R1-Searcher-plus.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. Builds on 23 of its 33 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

23 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025AirRAG: Autonomous Strategic Planning and Reasoning Steer Retrieval Augmented Generation17 Jan

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  10. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  11. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  12. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
10 more
  1. 2024Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexitycited by 437
  2. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  3. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 343
  4. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  5. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 165
  6. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  7. 2025Atom of Thoughts for Markov LLM Test-Time Scalingcited by 54
  8. 2024Long Term Memory: The Foundation of AI Self-Evolutioncited by 29
  9. 2024OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domaincited by 28
  10. 2024Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Taskscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 10.

30 papers.

Sort:

By area:

  1. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 66
  5. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 43
  6. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 27
  7. 2025Towards Effective Code-Integrated Reasoningcited by 23
  8. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  9. 2025GTM: Simulating the World of Tools for AI Agentscited by 12
  10. 2025Resisting Contextual Interference in RAG via Parametric-Knowledge Reinforcementcited by 11
  11. 2026ClawGym: A Scalable Framework for Building Effective Claw Agentscited by 11
  12. 2025VeriWeb: Verifiable Long-Chain Web Benchmark for Agentic Information-Seekingcited by 10
18 more
  1. 2025ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewardscited by 10
  2. 2025HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generationcited by 9
  3. 2025EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notescited by 8
  4. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  5. 2026GraphSearch: Agentic Search-Augmented Reasoning for Zero-Shot Graph Learningcited by 7
  6. 2026DLLM-Searcher: Adapting Diffusion Large Language Model for Search Agentscited by 7
  7. 2026JADE: Bridging the Strategic-Operational Gap in Dynamic Agentic RAGcited by 6
  8. 2025RE-Searcher: Robust Agentic Search with Goal-oriented Planning and Self-reflectioncited by 5
  9. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoningcited by 4
  10. 2026ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generationcited by 3
  11. 2025From Evidence to Trajectory: Abductive Reasoning Path Synthesis for Retrieval-Augmented Generation Agents Developmentcited by 2
  12. 2026Experience as a Compass: Multi-agent RAG with Evolving Orchestration and Agent Promptscited by 2
  13. 2026Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agentscited by 2
  14. 2025Agentic-KGR: Co-evolutionary Knowledge Graph Construction through Multi-Agent Reinforcement Learningcited by 1
  15. 2025Proactive Reasoning-with-Retrieval Framework for Medical Multimodal Large Language Modelscited by 1
  16. 2026IntentRL: Training Proactive User-intent Agents for Open-ended Deep Research via Reinforcement Learningcited by 1
  17. 2026SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Bankscited by 0
  18. 2026Test-Time Deep Thinking to Explore Implicit Rulescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
  2. 2025MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval
  3. 2025RAG-R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism
  4. 2025Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent
  5. 2025R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning
  6. 2025R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
  7. 2025UR2: Unify RAG and Reasoning through Reinforcement Learning
  8. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents