# Benchmarking LLM Tool-Use in the Wild

arXiv: https://arxiv.org/abs/2604.06185 · HTML: https://mlsanity.com/paper/2604.06185/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 13 Feb 2026.

## Builds on (most cited first, 19 of 19)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models](https://mlsanity.com/paper/2403.07714/) (2024)
- [AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls](https://mlsanity.com/paper/2402.04253/) (2024)
- [T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step](https://mlsanity.com/paper/2312.14033/) (2023)
- [Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios](https://mlsanity.com/paper/2401.17167/) (2024)
- [Direct Multi-Turn Preference Optimization for Language Agents](https://mlsanity.com/paper/2406.14868/) (2024)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [Improving Multi-turn Dialogue Modelling with Utterance ReWriter](https://mlsanity.com/paper/1906.07004/) (2019)
- [Hammer: Robust Function-Calling for On-Device Language Models via Function Masking](https://mlsanity.com/paper/2410.04587/) (2024)
- [Benchmarking Agentic Workflow Generation](https://mlsanity.com/paper/2410.07869/) (2024)
- [Exploring the Necessity of Reasoning in LLM-based Agent Scenarios](https://mlsanity.com/paper/2503.11074/) (2025)

## Built on by (most cited first, 17 of 17)

- [The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration](https://mlsanity.com/paper/2603.22862/) (2026)
- [Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents](https://mlsanity.com/paper/2608.13179/) (2026)
- [FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance](https://mlsanity.com/paper/2607.19409/) (2026)
- [AllocBench: Measuring Online Tool Allocation Capability in LLM Agents](https://mlsanity.com/paper/2607.23332/) (2026)
- [Execution-First Synthetic Tool-Use Trace Generation for LLM Agents](https://mlsanity.com/paper/2607.29175/) (2026)
- [Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions](https://mlsanity.com/paper/2606.03318/) (2026)
- [Blockchain Empowered Trustworthy Agent Networks: Foundations, Taxonomy, and Future Directions](https://mlsanity.com/paper/2608.04626/) (2026)
- [Lomekwi: Resource-Bounded Tool Discovery in LLM Agents](https://mlsanity.com/paper/2607.16961/) (2026)
- [DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents](https://mlsanity.com/paper/2608.10037/) (2026)
- [UserToolBench: A User-Profile-Hidden Benchmark for Personalized Decision Making in Tool-Use LLMs](https://mlsanity.com/paper/2608.10042/) (2026)
- [ATLAS: Dual-Horizon Diagnostic Evaluation for Industrial Tool-Use Agents](https://mlsanity.com/paper/2608.30685/) (2026)
- [RecToolBench: Benchmarking Recommendation-Specific Tool Orchestration under Fuzzy User Intent](https://mlsanity.com/paper/2609.30717/) (2026)
- [Toward Interactive Understanding of Code APIs](https://mlsanity.com/paper/2609.32081/) (2026)
- [TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces](https://mlsanity.com/paper/2609.33295/) (2026)
- [Asking for What Was Never Requested: Horizontal and Vertical Proactivity in Agents](https://mlsanity.com/paper/2609.37236/) (2026)
- [Do Tool Calls Execute as Intended? Measuring and Repairing Intent-Execution Correspondence in LLM Agents](https://mlsanity.com/paper/2610.04375/) (2026)
- [JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents](https://mlsanity.com/paper/2610.12957/) (2026)

## Abstract (arXiv)

Fulfilling user needs through Large Language Model multi-turn, multi-step tool-use is rarely a straightforward process. Real user interactions are inherently wild, being intricate, messy, and flexible. We identify three key challenges from user behaviour: compositional tasks that demand efficient orchestration of tool-call topologies, implicit intent spread across dialogue turns that require contextual inference, and instruction transition, which mixes task queries, clarifications, and casual conversation, forcing LLMs to adjust their policies on the fly. Existing benchmarks overlook these behaviors, making the apparent progress of LLMs on tool-use spurious. To address this, we introduce WildToolBench, an LLM tool-use benchmark grounded in real-world user behavior patterns. Comprehensive evaluations of 57 LLMs reveal that no model achieves an accuracy of more than 15%, indicating a substantial gap in the robustness of LLMs' agentic ability. Controlled experiments and in-depth analyses further indicate that the real challenge for LLM tool-use lies not in artificially complex tasks, but in the wild nature of user behavior, emphasizing the need to reconsider the interactions among LLMs, users, and tools.