# ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

arXiv: https://arxiv.org/abs/2607.04686 · HTML: https://mlsanity.com/paper/2607.04686/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 17 of 17)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents](https://mlsanity.com/paper/2407.18901/) (2024)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models](https://mlsanity.com/paper/2403.07714/) (2024)
- [The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism](https://mlsanity.com/paper/2407.10457/) (2024)
- [Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks](https://mlsanity.com/paper/2508.00828/) (2025)
- [MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents](https://mlsanity.com/paper/2501.14654/) (2025)
- [SMART: Self-Aware Agent for Tool Overuse Mitigation](https://mlsanity.com/paper/2502.11435/) (2025)
- [ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models](https://mlsanity.com/paper/2406.20015/) (2024)

## Built on by (most cited first, 11 of 11)

- [Execution-First Synthetic Tool-Use Trace Generation for LLM Agents](https://mlsanity.com/paper/2607.29175/) (2026)
- [Interface-Induced Trajectory Censoring](https://mlsanity.com/paper/2609.03966/) (2026)
- [Policy-Masked Private Experts: Auditable and Reversible Capability Access Control in Sparse MoE Models](https://mlsanity.com/paper/2608.06690/) (2026)
- [Source-Dependent Deference in Medical Imaging Agents Under Falsified Findings: A Pilot Audit](https://mlsanity.com/paper/2608.29800/) (2026)
- [Fabrication After Tool Failure: Tool-Augmented Agents Assert Values Their Tools Did Not Return](https://mlsanity.com/paper/2609.14758/) (2026)
- [BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents](https://mlsanity.com/paper/2609.16305/) (2026)
- [What Stops a Small Language Model From Driving a Database Agent](https://mlsanity.com/paper/2609.21341/) (2026)
- [Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse](https://mlsanity.com/paper/2609.26836/) (2026)
- [Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agents](https://mlsanity.com/paper/2610.02627/) (2026)
- [One Step at a Time: Trading LLM Autonomy for Process Predictability](https://mlsanity.com/paper/2610.07817/) (2026)
- [AgentSWE: Can Coding Agents Build the Agent You Actually Want?](https://mlsanity.com/paper/2610.13361/) (2026)

## Abstract (arXiv)

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.