mlsanityData to 14 Oct 2026

StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning

arXiv 2410.07745 ↗First posted 10 Oct 2024cs.CLcited by 25

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Despite their powerful text generation capabilities, large language models (LLMs) still struggle to effectively utilize external tools to solve complex tasks, a challenge known as tool learning. Existing methods primarily rely on supervised fine-tuning, treating tool learning as a text generation problem while overlooking the decision-making complexities inherent in multi-step contexts. In this work, we propose modeling tool learning as a dynamic decision-making process and introduce StepTool, a novel step-grained reinforcement learning framework that enhances LLMs' capabilities in multi-step tool use. StepTool comprises two key components: Step-grained Reward Shaping, which assigns rewards to each tool interaction based on its invocation success and contribution to task completion; and Step-grained Optimization, which applies policy gradient methods to optimize the model across multiple decision steps. Extensive experiments across diverse benchmarks show that StepTool consistently outperforms both SFT-based and RL-based baselines in terms of task Pass Rate and Recall of relevant tools. Furthermore, our analysis suggests that StepTool helps models discover new tool-use strategies rather than merely re-weighting prior knowledge. These results highlight the importance of fine-grained decision modeling in tool learning and establish StepTool as a general and robust solution for enhancing multi-step tool use in LLMs. Code and data are available at https://github.com/yuyq18/StepTool.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 36 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

28 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Reinforcing Language Agents via Policy Optimization with Action Decomposition23 May
  2. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents4 Mar
  3. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL29 Feb

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
13 more
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  3. 2023Tool Learning with Foundation Modelscited by 302
  4. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  5. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  6. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  7. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoningcited by 102
  8. 2024ToolNet: Connecting Large Language Models with Massive Tools via Tool Graphcited by 60
  9. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  10. 2024ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrievalcited by 41
  11. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  12. 2024True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learningcited by 36
  13. 2024Towards Completeness-Oriented Tool Retrieval for Large Language Modelscited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 17.

25 papers.

Sort:

By area:

  1. 2025Agentic Reinforced Policy Optimizationcited by 172
  2. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 68
  3. 2025Learning to Orchestrate Agents in Natural Language with the Conductorcited by 23
  4. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Modelscited by 17
  5. 2026ToolGate: Contract-Grounded and Verified Tool Execution for LLMscited by 15
  6. 2026CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Usecited by 14
  7. 2025Hybrid Reward Normalization for Process-supervised Non-verifiable Agentic Taskscited by 10
  8. 2026PhGPO: Pheromone-Guided Policy Optimization for Long-Horizon Tool Planningcited by 9
  9. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  10. 2026DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Trainingcited by 5
  11. 2025PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awarenesscited by 4
  12. 2026ATLAS: Agentic Test-time Learning-to-Allocate Scalingcited by 4
13 more
  1. 2025ML-Tool-Bench: Tool-Augmented Planning for ML Taskscited by 3
  2. 2026Teaching LLMs to Learn Tool Trialing and Execution through Environment Interactioncited by 3
  3. 2026RLVP: Penalize the Path, Reward the Outcomecited by 2
  4. 2026Synthesize and Reward -- Reinforcement Learning for Multi-Step Tool Use in Live Environmentscited by 1
  5. 2026Federated Agentic AI for Wireless Networks: Fundamentals, Approaches, and Applicationscited by 0
  6. 2026HTAA: Enhancing LLM Planning via Hybrid Toolset Agentization & Adaptationcited by 0
  7. 2026Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learningcited by 0
  8. 2026Capability-Aligned Hierarchical Learning for Tool-Augmented LLMscited by 0
  9. 2026ToolLIFT: Lifting Tool-Specific Trajectories into Function-Level Graphs for Generalizable Tool Planningcited by 0
  10. 2026Toolcompass: Guiding Tool Trialing, Not Suppressing Itcited by 0
  11. 2026Choosing Before Acting: Comparative Value Estimation for Long-Horizon Tool-Use Agentscited by 0
  12. 2026CIPO: Counterfactual Imagination Policy Optimization for Adaptive Tool Granularity Selectioncited by 0
  13. 2026MADS: Multi-Agent Dialog Simulation for Robust Conversational Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ToolRL: Reward is All Tool Learning Needs
  2. 2025Step-wise Policy for Rare-tool Knowledge (SPaRK): Offline RL that Drives Diverse Tool Use in LLMs
  3. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments
  4. 2026ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning
  5. 2025ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMs
  6. 2026Joint Optimization of Tool Creation and Use for Large Language Model Agents
  7. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  8. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch