mlsanityData to 14 Oct 2026

Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios

arXiv 2401.17167 ↗First posted 30 Jan 2024cs.CLcited by 86

Not on the map. Why?

Cited by 52 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent trend of using Large Language Models (LLMs) as tool agents in real-world applications underscores the necessity for comprehensive evaluations of their capabilities, particularly in complex scenarios involving planning, creating, and using tools. However, existing benchmarks typically focus on simple synthesized queries that do not reflect real-world complexity, thereby offering limited perspectives in evaluating tool utilization. To address this issue, we present UltraTool, a novel benchmark designed to improve and evaluate LLMs' ability in tool utilization within real-world scenarios. UltraTool focuses on the entire process of using tools - from planning and creating to applying them in complex tasks. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage and simplifies the task solving by mapping out the intermediate steps. Thus, unlike previous work, it eliminates the restriction of pre-defined toolset. Through extensive experiments on various LLMs, we offer novel insights into the evaluation of capabilities of LLMs in tool utilization, thereby contributing a fresh perspective to this rapidly evolving field. The benchmark is publicly available at https://github.com/JoeYing1019/UltraTool.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 14 of its 26 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

14 of 26 references.

References · 14

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  6. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  7. 2023Augmented Language Models: a Surveycited by 403
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  9. 2023Large Language Models as Tool Makerscited by 303
  10. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  11. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  12. 2022TALM: Tool Augmented Language Modelscited by 197
2 more
  1. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 55
  2. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 45; 2026: 26.

86 papers.

Sort:

By area:

  1. 2025Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directionscited by 265
  2. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  3. 2024Tool Learning with Large Language Models: A Surveycited by 223
  4. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  5. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  6. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 104
  7. 2024A Survey on Self-Evolution of Large Language Modelscited by 91
  8. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  9. 2025Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistantcited by 54
  10. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 50
  11. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  12. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 47
38 more
  1. 2024Can Graph Learning Improve Planning in LLM-based Agents?cited by 43
  2. 2026τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledgecited by 39
  3. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  4. 2025CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agentscited by 33
  5. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  6. 2024Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generationcited by 27
  7. 2025UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarioscited by 26
  8. 2024AppBench: Planning of Multiple APIs from Various APPs for Complex User Instructioncited by 24
  9. 2025Procedural Environment Generation for Tool-Use Agentscited by 24
  10. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  11. 2025How Far are LLMs from Being Our Digital Twins? A Benchmark for Persona-Based Behavior Chain Simulationcited by 22
  12. 2025ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filteringcited by 19
  13. 2025Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systemscited by 18
  14. 2024Enhancing Tool Retrieval with Iterative Feedback from Large Language Modelscited by 17
  15. 2026Benchmarking LLM Tool-Use in the Wildcited by 17
  16. 2025Are Your Agents Upward Deceivers?cited by 16
  17. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
  18. 2024Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translationcited by 14
  19. 2025StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIscited by 14
  20. 2026ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Contextcited by 14
  21. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missionscited by 12
  22. 2025Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challengescited by 12
  23. 2025Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactionscited by 12
  24. 2025GenTool: Enhancing Tool Generalization in Language Models through Zero-to-One and Weak-to-Strong Simulationcited by 10
  25. 2025Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Servicescited by 10
  26. 2025PhysToolBench: Benchmarking Physical Tool Understanding for MLLMscited by 9
  27. 2024Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?cited by 8
  28. 2025TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environmentscited by 8
  29. 2026CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertaintycited by 8
  30. 2024Tulip Agent -- Enabling LLM-Based Agents to Solve Tasks Using Large Tool Librariescited by 7
  31. 2025ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolutioncited by 7
  32. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interactioncited by 7
  33. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6
  34. 2025ToolSpectrum : Towards Personalized Tool Utilization for Large Language Modelscited by 6
  35. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  36. 2025FamilyTool: A Multi-hop Personalized Tool Use Benchmarkcited by 5
  37. 2025ToolHaystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactionscited by 5
  38. 2025Actions Speak Louder than Prompts: A Large-Scale Study of LLMs for Graph Inferencecited by 5

50 most cited of 86 All 86 (JSON)