mlsanityData to 14 Oct 2026

ToolRL: Reward is All Tool Learning Needs

arXiv 2504.13958 ↗First posted 16 Apr 2025cs.LG cs.AI cs.CLcited by 362

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Current Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities. However, SFT struggles to generalize to unfamiliar or complex tool use scenarios. Recent advancements in reinforcement learning (RL), particularly with R1-like models, have demonstrated promising reasoning and generalization abilities. Yet, reward design for tool use presents unique challenges: multiple tools may be invoked with diverse parameters, and coarse-grained reward signals, such as answer matching, fail to offer the finegrained feedback required for effective learning. In this work, we present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm. We systematically explore a wide range of reward strategies, analyzing their types, scales, granularity, and temporal dynamics. Building on these insights, we propose a principled reward design tailored for tool use tasks and apply it to train LLMs using Group Relative Policy Optimization (GRPO). Empirical evaluations across diverse benchmarks demonstrate that our approach yields robust, scalable, and stable training, achieving a 17% improvement over base models and a 15% gain over SFT models. These results highlight the critical role of thoughtful reward design in enhancing the tool use capabilities and generalization performance of LLMs. All the codes are released to facilitate future research.

Key facts. Cited by 362 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 63 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

54 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025ToRL: Scaling Tool-Integrated RL30 Mar
  2. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models14 Sep
  3. 2023Rational Decision-Making Agent with Internalized Utility Judgment24 Aug

Other references · 51

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
38 more
  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  5. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  6. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  7. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  8. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  9. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  10. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  11. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  12. 2023Tool Learning with Foundation Modelscited by 302
  13. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  14. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  15. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  16. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  17. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  18. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  19. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  20. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  21. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 210
  22. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  23. 2025LIMR: Less is More for RL Scalingcited by 139
  24. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  25. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  26. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  27. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 111
  28. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 100
  29. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 93
  30. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 92
  31. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  32. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn'tcited by 74
  33. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Maskingcited by 55
  34. 2024BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experimentscited by 53
  35. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  36. 2023Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Modelcited by 29
  37. 2025Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Modelcited by 20
  38. 2025T1: Tool-integrated Verification for Test-time Compute Scaling in Small Language Modelscited by 13

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 130; 2026: 232.

362 papers.

Sort:

By area:

  1. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  2. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  3. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 237
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  5. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
  6. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 188
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  8. 2025Agentic Reinforced Policy Optimizationcited by 172
  9. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 130
  10. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 127
  11. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  12. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
38 more
  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2025Agent Learning via Early Experiencecited by 85
  3. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  4. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
  5. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 79
  6. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  7. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 78
  8. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 77
  9. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 76
  10. 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learningcited by 72
  11. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 72
  12. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  13. 2025Scaling Test-time Compute for LLM Agentscited by 63
  14. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 62
  15. 2025Visual Agentic Reinforcement Fine-Tuningcited by 55
  16. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  17. 2025Towards General Agentic Intelligence via Environment Scalingcited by 54
  18. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  19. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 51
  20. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  21. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  22. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 49
  23. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 47
  24. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 44
  25. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  26. 2025Agentic Entropy-Balanced Policy Optimizationcited by 40
  27. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agentcited by 39
  28. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  29. 2025Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoningcited by 35
  30. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
  31. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 33
  32. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 33
  33. 2025We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoningcited by 32
  34. 2025MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool usecited by 32
  35. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 32
  36. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 31
  37. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 31
  38. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 30

50 most cited of 362 All 362 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  2. 2026ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning
  3. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning
  4. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training
  5. 2025Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning
  6. 2026Joint Optimization of Tool Creation and Use for Large Language Model Agents
  7. 2023Tool-Augmented Reward Modeling
  8. 2025ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMs