mlsanityData to 7 Oct 2026

Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning

arXiv 2503.09516 ↗First posted 12 Mar 2025cs.CL cs.AI cs.IRcited by 1,371

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Efficiently acquiring external knowledge and up-to-date information is essential for effective reasoning and text generation in large language models (LLMs). Prompting advanced LLMs with reasoning capabilities to use search engines during inference is often suboptimal, as the LLM might not fully possess the capability on how to interact optimally with the search engine. This paper introduces Search-R1, an extension of reinforcement learning (RL) for reasoning frameworks where the LLM learns to autonomously generate (multiple) search queries during step-by-step reasoning with real-time retrieval. Search-R1 optimizes LLM reasoning trajectories with multi-turn search interactions, leveraging retrieved token masking for stable RL training and a simple outcome-based reward function. Experiments on seven question-answering datasets show that Search-R1 improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines under the same setting. This paper further provides empirical insights into RL optimization methods, LLM choices, and response length dynamics in retrieval-augmented reasoning. The code and model checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.

Key facts. Cited by 1,371 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 53 of its 64 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

53 of 64 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Search-o1: Agentic Search-Enhanced Large Reasoning Models9 Jan
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  3. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions20 Dec

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2024Qwen2.5 Technical Reportcited by 4,713
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
38 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  11. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  13. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  14. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  15. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  16. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  17. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  18. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  19. 2023Active Retrieval Augmented Generationcited by 506
  20. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  21. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  22. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  23. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  24. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  25. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  26. 2024Iterative Reasoning Preference Optimizationcited by 244
  27. 2024Reinforcement learningcited by 236
  28. 2023Large Language Models in Finance: A Surveycited by 223
  29. 2024Tool Learning with Large Language Models: A Surveycited by 221
  30. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  31. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  32. 2022Dense Text Retrieval based on Pretrained Language Models: A Surveycited by 136
  33. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 122
  34. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  35. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
  36. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  37. 2024RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generationcited by 20
  38. 2024Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrievalcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 511; 2026: 860.

1,371 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  3. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  5. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  6. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  7. 2025Memory in the Age of AI Agentscited by 285
  8. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  9. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 252
  10. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  11. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  12. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
38 more
  1. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 218
  2. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  3. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  5. 2025BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agentcited by 183
  6. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  8. 2025Agentic Reinforced Policy Optimizationcited by 166
  9. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  10. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  11. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  12. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  13. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  14. 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 136
  15. 2025ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarizationcited by 132
  16. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  17. 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 129
  18. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agentscited by 126
  19. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  20. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  21. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  22. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  23. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  24. 2025WebShaper: Agentically Data Synthesizing via Information-Seeking Formalizationcited by 118
  25. 2025Reinforcement Learning with Rubric Anchorscited by 108
  26. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  27. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  28. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  29. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  30. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  31. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  32. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 82
  33. 2025Agent Learning via Early Experiencecited by 79
  34. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  35. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 77
  36. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  37. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 75
  38. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73

50 most cited of 1,371 All 1,371 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
  2. 2025SEM: Reinforcement Learning for Search-Efficient Large Language Models
  3. 2025On the Emergence of Thinking in LLMs I: Searching for the Right Intuition
  4. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  5. 2025ReZero: Enhancing LLM search ability by trying one-more-time
  6. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  7. 2025Outcome-based Exploration for LLM Reasoning
  8. 2025Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents