mlsanityData to 7 Oct 2026

s3: You Don't Need That Much Data to Train a Search Agent via RL

arXiv 2505.14146 ↗First posted 20 May 2025cs.AI cs.CLcited by 41

On the map: Retrieval-augmented generation → Agentic RAG, agents and memory

also in 1 more
Abstract (arXiv)

Retrieval-augmented generation (RAG) systems empower large language models (LLMs) to access external knowledge during inference. Recent advances have enabled LLMs to act as search agents via reinforcement learning (RL), improving information acquisition through multi-turn interactions with retrieval engines. However, existing approaches either optimize retrieval using search-only metrics (e.g., NDCG) that ignore downstream utility or fine-tune the entire LLM to jointly reason and retrieve-entangling retrieval with generation and limiting the real search utility and compatibility with frozen or proprietary models. In this work, we propose s3, a lightweight, model-agnostic framework that decouples the searcher from the generator and trains the searcher using a Gain Beyond RAG reward: the improvement in generation accuracy over naive RAG. s3 requires only 2.4k training samples to outperform baselines trained on over 70x more data, consistently delivering stronger downstream performance across six general QA and five medical QA benchmarks.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 41 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

28 of 41 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  11. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
16 more
  1. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  3. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  6. 2019Passage Re-ranking with BERTcited by 629
  7. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  8. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Modelscited by 423
  9. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  10. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  11. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  12. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  13. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150
  14. 2021A Replication Study of Dense Passage Retrievercited by 37
  15. 2025SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reductioncited by 24
  16. 2024Retrieve-Plan-Generation: An Iterative Planning and Answering Framework for Knowledge-Intensive LLM Generationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 24.

41 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 58
  3. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 39
  4. 2025MedResearcher-R1: Expert-Level Medical Deep Researcher via A Knowledge-Informed Trajectory Synthesis Frameworkcited by 22
  5. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  6. 2025DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learningcited by 14
  7. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  8. 2025ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewardscited by 10
  9. 2025AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learningcited by 8
  10. 2025QAgent: A modular Search Agent with Interactive Query Understandingcited by 8
  11. 2026The Interplay of Harness Design and Post-Training in LLM Agentscited by 8
  12. 2025TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Frameworkcited by 7
29 more
  1. 2025AutoGraph-R1: End-to-End Reinforcement Learning for Knowledge Graph Constructioncited by 6
  2. 2026JADE: Bridging the Strategic-Operational Gap in Dynamic Agentic RAGcited by 6
  3. 2025Uncertainty Quantification for Retrieval-Augmented Reasoningcited by 5
  4. 2025Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agentscited by 4
  5. 2025Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metricscited by 4
  6. 2025Progressive Multimodal Search and Reasoning for Knowledge-Intensive Visual Question Answeringcited by 3
  7. 2026M3Searcher: Modular Multimodal Information Seeking Agency with Retrieval-Oriented Reasoningcited by 3
  8. 2026BranPO: Scalable Contrastive Branch Sampling for Long-Horizon Agentic Reinforcement Learningcited by 3
  9. 2025Structure-Augmented Reasoning Generationcited by 2
  10. 2026Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilitiescited by 2
  11. 2026MultiCube-RAG for Multi-hop Question Answeringcited by 2
  12. 2026LensVLM: Selective Context Expansion for Compressed Visual Representation of Textcited by 2
  13. 2026SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Searchcited by 2
  14. 2026ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agentscited by 2
  15. 2026Divide and Cooperate: Role-Decomposed Multi-Agent LLM Training with Cross-Agent Learning Signalscited by 2
  16. 2026Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?cited by 2
  17. 2025Multi-hop Reasoning via Early Knowledge Alignmentcited by 1
  18. 2026Plan Before Search: Search Agents Need Plancited by 1
  19. 2026It Takes Two to Match: Co-Evolving Generative Retriever with Reinforcement Learningcited by 1
  20. 2026Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusioncited by 0
  21. 2026Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Searchcited by 0
  22. 2026SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Bankscited by 0
  23. 2026DeepRefine: Agentic Knowledge Refinement via Reinforcement Learningcited by 0
  24. 2026COMPASS: Cognitive MCTS-Guided Process Alignment for Safe Search Agentscited by 0
  25. 2026HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learningcited by 0
  26. 2026MCite-RL: Towards Reliable Multimodal RAG via Citation-enhanced Agentic Reinforcement Learningcited by 0
  27. 2026FORGE: Form-Optimal Routing of Grounded Evidence for Frozen LLM Agentscited by 0
  28. 2026BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learningcited by 0
  29. 2026False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing Retrieval for RAG via Reinforcement Learning
  2. 2025R3-RAG: Learning Step-by-Step Reasoning and Retrieval for LLMs via Reinforcement Learning
  3. 2025MARAG-R1: Beyond Single Retriever via Reinforcement-Learned Multi-Tool Agentic Retrieval
  4. 2025C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
  5. 2026OThink-SRR1: Search, Refine and Reasoning with Reinforced Learning for Large Language Models
  6. 2025Search Wisely: Mitigating Sub-optimal Agentic Searches By Reducing Uncertainty
  7. 2025R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning
  8. 2025Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation