mlsanityData to 7 Oct 2026

MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use

arXiv 2310.03128 ↗First posted 4 Oct 2023cs.SE cs.CLcited by 209

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities. Recently, many studies have focused on the tool utilization ability of LLMs. They primarily investigated how LLMs effectively collaborate with given specific tools. However, in scenarios where LLMs serve as intelligent agents, as seen in applications like AutoGPT and MetaGPT, LLMs are expected to engage in intricate decision-making processes that involve deciding whether to employ a tool and selecting the most suitable tool(s) from a collection of available tools to fulfill user requests. Therefore, in this paper, we introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools. Specifically, we create a dataset called ToolE within the benchmark. This dataset contains various types of user queries in the form of prompts that trigger LLMs to use tools, including both single-tool and multi-tool scenarios. Subsequently, we set the tasks for both tool usage awareness and tool selection. We define four subtasks from different perspectives in tool selection, including tool selection with similar choices, tool selection in specific scenarios, tool selection with possible reliability issues, and multi-tool selection. We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools, highlighting the existing gaps between LLMs and genuine intelligent agents. However, through the error analysis, we found there is still significant room for improvement. Finally, we conclude with insights for tool developers -- we strongly recommend that tool developers choose an appropriate rewrite model for generating new descriptions based on the downstream LLM the tool will apply to. Our code is in https://github.com/HowieHwong/MetaTool.

Key facts. Cited by 209 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 44 of its 84 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

44 of 84 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  3. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  4. 2023Large Language Models as Tool Makers26 May
  5. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  6. 2023Tool Learning with Foundation Models17 Apr
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  7. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2023BloombergGPT: A Large Language Model for Financecited by 923
25 more
  1. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  2. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  3. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  4. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  5. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  6. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  7. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  8. 2023FinGPT: Open-Source Financial Large Language Modelscited by 244
  9. 2023Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender Systemcited by 241
  10. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  11. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  12. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  13. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  14. 2023HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledgecited by 161
  15. 2023Uncovering ChatGPT's Capabilities in Recommender Systemscited by 160
  16. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  17. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  18. 2022Internet-augmented language models through few-shot prompting for open-domain question answeringcited by 131
  19. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  20. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  21. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  22. 2023ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluationcited by 85
  23. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  24. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  25. 2023FoodGPT: A Large Language Model in Food Testing Domain with Incremental Pre-training and Knowledge Graph Promptcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 58; 2025: 70; 2026: 77.

209 papers.

Sort:

By area:

  1. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  2. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  4. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  5. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  6. 2024Tool Learning with Large Language Models: A Surveycited by 221
  7. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  8. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  9. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  10. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  11. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
38 more
  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  3. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  4. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
  5. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  6. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  7. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 66
  8. 2025ACEBench: Who Wins the Match Point in Tool Usage?cited by 66
  9. 2024MindSearch: Mimicking Human Minds Elicits Deep AI Searchercited by 65
  10. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 60
  11. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 59
  12. 2026SkillRouter: Skill Routing for LLM Agents at Scalecited by 57
  13. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaborationcited by 56
  14. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 56
  15. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  16. 2024Learning to Use Tools via Cooperative and Interactive Agentscited by 54
  17. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53
  18. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  19. 2024Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrievalcited by 50
  20. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 48
  21. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Modelscited by 48
  22. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Settingcited by 47
  23. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  24. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 46
  25. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Serverscited by 46
  26. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 45
  27. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applicationscited by 45
  28. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMscited by 39
  29. 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
  30. 2024Adaptive In-conversation Team Building for Language Model Agentscited by 38
  31. 2024ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agentscited by 36
  32. 2025When2Call: When (not) to Call Toolscited by 36
  33. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  34. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  35. 2024Efficient Tool Use with Chain-of-Abstraction Reasoningcited by 33
  36. 2026CUA-Skill: Develop Skills for Computer Using Agentcited by 32
  37. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  38. 2023Cooperation on the Fly: Exploring Language Agents for Ad Hoc Teamwork in the Avalon Gamecited by 27

50 most cited of 209 All 209 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MetaToolAgent: Towards Generalizable Tool Usage in LLMs through Meta-Learning
  2. 2025ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering
  3. 2024LLM With Tools: A Survey
  4. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
  5. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
  6. 2024MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation
  7. 2025BiasBusters: Uncovering and Mitigating Tool Selection Bias in Large Language Models
  8. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Models