mlsanityData to 14 Oct 2026

Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use

arXiv 2509.12867 ↗First posted 16 Sep 2025cs.LG cs.CVcited by 16

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large language models (LLMs) have demonstrated strong capabilities in language understanding and reasoning, yet they remain limited when tackling real-world tasks that require up-to-date knowledge, precise operations, or specialized tool use. To address this, we propose Tool-R1, a reinforcement learning framework that enables LLMs to perform general, compositional, and multi-step tool use by generating executable Python code. Tool-R1 supports integration of user-defined tools and standard libraries, with variable sharing across steps to construct coherent workflows. An outcome-based reward function, combining LLM-based answer judgment and code execution success, guides policy optimization. To improve training efficiency, we maintain a dynamic sample queue to cache and reuse high-quality trajectories, reducing the overhead of costly online sampling. Experiments on the GAIA benchmark show that Tool-R1 substantially improves both accuracy and robustness, achieving about 10% gain over strong baselines, with larger improvements on complex multi-step tasks. These results highlight the potential of Tool-R1 for enabling reliable and efficient tool-augmented reasoning in real-world applications. Our code will be available at https://github.com/YBYBZhang/Tool-R1.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 43 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

29 of 43 references.

References · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  12. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
17 more
  1. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  4. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  5. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  6. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  8. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
  9. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  10. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  11. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  12. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  13. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  14. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 93
  15. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 74
  16. 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
  17. 2025DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learningcited by 66

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 85
  2. 2025Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skillscited by 22
  3. 2026MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baselinecited by 20
  4. 2026Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agentscited by 5
  5. 2026Reasoning While Asking: Transforming Reasoning Large Language Models from Passive Solvers to Proactive Inquirerscited by 3
  6. 2025RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learningcited by 2
  7. 2026Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoningcited by 2
  8. 2026Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversationscited by 2
  9. 2026Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RLcited by 2
  10. 2026Synthesize and Reward -- Reinforcement Learning for Multi-Step Tool Use in Live Environmentscited by 1
  11. 2026SQLConductor: Search-to-Policy Learning for Step-wise Text-to-SQL Orchestrationcited by 1
  12. 2026Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perceptioncited by 0
4 more
  1. 2026A Task-Driven Evaluation of UAV Detection and Tracking under Synthetic Fogcited by 0
  2. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agentscited by 0
  3. 2026SCOUT: Self-Checking and Recovery-Aware Tool-Thought Agents for Ultra-Long Egocentric Video Reasoningcited by 0
  4. 2026CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
  2. 2026ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning
  3. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments
  4. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
  5. 2025ToolRL: Reward is All Tool Learning Needs
  6. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
  7. 2025Enhancing Agentic RL with Progressive Reward Shaping and Value-based Sampling Policy Optimization
  8. 2026Joint Optimization of Tool Creation and Use for Large Language Model Agents