mlsanityData to 7 Oct 2026

What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks

arXiv 2407.03007 ↗First posted 3 Jul 2024cs.CL cs.AIcited by 4

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Tool learning methods have enhanced the ability of large language models (LLMs) to interact with real-world applications. Many existing works fine-tune LLMs or design prompts to enable LLMs to select appropriate tools and correctly invoke them to meet user requirements. However, it is observed in previous works that the performance of tool learning varies from tasks, datasets, training settings, and algorithms. Without understanding the impact of these factors, it can lead to inconsistent results, inefficient model deployment, and suboptimal tool utilization, ultimately hindering the practical integration and scalability of LLMs in real-world scenarios. Therefore, in this paper, we explore the impact of both internal and external factors on the performance of tool learning frameworks. Through extensive experiments on two benchmark datasets, we find several insightful conclusions for future work, including the observation that LLMs can benefit significantly from increased trial and exploration. We believe our empirical study provides a new perspective for future tool learning research.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 37 of its 50 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

37 of 50 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Learning to Use Tools via Cooperative and Interactive Agents5 Mar
  2. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages16 Feb
  3. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning16 Jan
  4. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  5. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum27 Aug
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  7. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  9. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  10. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  11. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  12. 2023Tool Learning with Foundation Models17 Apr
2 more
  1. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 23

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  4. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  5. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  6. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  7. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  8. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  9. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  10. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  11. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  12. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
11 more
  1. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  2. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  3. 2024Is Temperature the Creativity Parameter of Large Language Models?cited by 92
  4. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  5. 2023Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Modelscited by 37
  6. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 37
  7. 2022Robustness of Learning from Task Instructionscited by 34
  8. 2023Probing the Creativity of Large Language Models: Can models produce divergent semantic association?cited by 31
  9. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 30
  10. 2024Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Biascited by 28
  11. 2023Robust Prompt Optimization for Large Language Models Against Distribution Shiftscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025TTPA: Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluationcited by 9
  2. 2025ToolCoder: A Systematic Code-Empowered Tool Learning Framework for Large Language Modelscited by 8
  3. 2026From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skillscited by 5
  4. 2026PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Tool Learning with Large Language Models: A Survey
  2. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
  3. 2024Learning Evolving Tools for Large Language Models
  4. 2025ToLeaP: Rethinking Development of Tool Learning with Large Language Models
  5. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
  6. 2025Alignment for Efficient Tool Calling of Large Language Models
  7. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions
  8. 2025ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolution