# Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction

arXiv: https://arxiv.org/abs/2601.12762 · HTML: https://mlsanity.com/paper/2601.12762/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 19 Jan 2026.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 21 of 21)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs](https://mlsanity.com/paper/2303.16434/) (2023)
- [ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use](https://mlsanity.com/paper/2501.02506/) (2025)
- [Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees](https://mlsanity.com/paper/2406.07115/) (2024)
- [StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning](https://mlsanity.com/paper/2410.07745/) (2024)
- [TOOLVERIFIER: Generalization to New Tools via Self-Verification](https://mlsanity.com/paper/2402.14158/) (2024)
- [Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning](https://mlsanity.com/paper/2506.04625/) (2025)
- [Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments](https://mlsanity.com/paper/2508.08791/) (2025)
- [INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair](https://mlsanity.com/paper/2311.09868/) (2023)
- [Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch](https://mlsanity.com/paper/2511.01934/) (2025)
- [CodeTool: Enhancing Programmatic Tool Invocation of LLMs via Process Supervision](https://mlsanity.com/paper/2503.20840/) (2025)

## Built on by (most cited first, 3 of 3)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [Toolcompass: Guiding Tool Trialing, Not Suppressing It](https://mlsanity.com/paper/2609.25678/) (2026)
- [Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints](https://mlsanity.com/paper/2610.18459/) (2026)

## Abstract (arXiv)

Equipping Large Language Models (LLMs) with external tools enables them to solve complex real-world problems. However, the robustness of existing methods remains a critical challenge when confronting novel or evolving tools. Existing trajectory-centric paradigms primarily rely on memorizing static solution paths during training, which limits the ability of LLMs to generalize tool usage to newly introduced or previously unseen tools. In this paper, we propose ToolMaster, a framework that shifts tool use from imitating golden tool-calling trajectories to actively learning tool usage through interaction with the environment. To optimize LLMs for tool planning and invocation, ToolMaster adopts a trial-and-execution paradigm, which trains LLMs to first imitate teacher-generated trajectories containing explicit tool trials and self-correction, followed by reinforcement learning to coordinate the trial and execution phases jointly. This process enables agents to autonomously explore correct tool usage by actively interacting with environments and forming experiential knowledge that benefits tool execution. Experimental results demonstrate that ToolMaster significantly outperforms existing baselines in terms of generalization and robustness across unseen or unfamiliar tools. All code and data are available at https://github.com/NEUIR/ToolMaster.