mlsanityData to 7 Oct 2026

StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models

arXiv 2403.07714 ↗First posted 12 Mar 2024cs.CLcited by 129

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 20 of its 34 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

20 of 34 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  2. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback19 Sep
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  4. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  5. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  6. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  7. 2023Large Language Models as Tool Makers26 May
  8. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  9. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  10. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 9

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  5. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  6. 2023Augmented Language Models: a Surveycited by 398
  7. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  8. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  9. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 42; 2026: 67.

129 papers.

Sort:

By area:

  1. 2025Kimi K2: Open Agentic Intelligencecited by 523
  2. 2024Tool Learning with Large Language Models: A Surveycited by 221
  3. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  4. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  5. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  6. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 72
  7. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  8. 2025Self-Challenging Language Model Agentscited by 56
  9. 2025Towards General Agentic Intelligence via Environment Scalingcited by 51
  10. 2025LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?cited by 47
  11. 2025Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agentscited by 42
  12. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 40
38 more
  1. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?cited by 40
  2. 2024Reducing Tool Hallucination via Reliability Alignmentcited by 37
  3. 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
  4. 2024Benchmarking Agentic Workflow Generationcited by 36
  5. 2025ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Modelscited by 34
  6. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  7. 2026ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditionscited by 31
  8. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  9. 2025MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool usecited by 30
  10. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
  11. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  12. 2025ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Datasetcited by 27
  13. 2024ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Modelscited by 25
  14. 2025Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learningcited by 22
  15. 2026Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agentscited by 22
  16. 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Casescited by 21
  17. 2025HCAST: Human-Calibrated Autonomy Software Taskscited by 19
  18. 2025NLI4VolVis: Natural Language Interaction for Volume Visualization via LLM Multi-Agents and Editable 3D Gaussian Splattingcited by 19
  19. 2024MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domainscited by 18
  20. 2025SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environmentscited by 18
  21. 2024ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMscited by 17
  22. 2026CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failurescited by 17
  23. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Modelscited by 16
  24. 2024Learning Evolving Tools for Large Language Modelscited by 15
  25. 2025LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queriescited by 15
  26. 2026Benchmarking LLM Tool-Use in the Wildcited by 15
  27. 2025ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memorycited by 14
  28. 2026Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Textcited by 13
  29. 2025Structured Agent Distillation for Large Language Modelcited by 11
  30. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  31. 2026EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RLcited by 11
  32. 2024Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?cited by 8
  33. 2025Information Density Principle for MLLM Benchmarkscited by 8
  34. 2025COMPASS: Benchmarking Constrained Optimization in LLM Agentscited by 8
  35. 2026BioAgent Bench: An AI Agent Evaluation Suite for Bioinformaticscited by 8
  36. 2026Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Usecited by 8
  37. 2026ToolFailBench: Diagnosing Tool-Use Failures in LLM Agentscited by 8
  38. 2026Agentic Test-Time Scaling for WebAgentscited by 7

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs
  2. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning
  3. 2024What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks
  4. 2025Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling
  5. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
  6. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use
  7. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
  8. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models