mlsanityData to 7 Oct 2026

ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios

arXiv 2401.00741 ↗First posted 1 Jan 2024cs.CL cs.AIcited by 35

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Existing evaluations of tool learning primarily focus on validating the alignment of selected tools for large language models (LLMs) with expected outcomes. However, these approaches rely on a limited set of scenarios where answers can be pre-determined, diverging from genuine needs. Furthermore, a sole emphasis on outcomes disregards the complex capabilities required for LLMs to effectively use tools. To tackle this issue, we propose ToolEyes, a fine-grained system tailored for the evaluation of the LLMs' tool learning capabilities in authentic scenarios. The system meticulously examines seven real-world scenarios, analyzing five dimensions crucial to LLMs in tool learning: format alignment, intent comprehension, behavior planning, tool selection, and answer organization. Additionally, ToolEyes incorporates a tool library boasting approximately 600 tools, serving as an intermediary between LLMs and the physical world. Evaluations involving ten LLMs across three categories reveal a preference for specific scenarios and limited cognitive abilities in tool learning. Intriguingly, expanding the model size even exacerbates the hindrance to tool learning. The code and data are available at https://github.com/Junjie-Ye/ToolEyes.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 26 of its 39 references that are in the map. First posted on arXiv 1 Jan 2024.

Builds on

26 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  3. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  4. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  5. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  6. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  7. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
6 more
  1. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  2. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138
  3. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  4. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 91
  5. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  6. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 14; 2026: 10.

35 papers.

Sort:

By area:

  1. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  2. 2024Tool Learning with Large Language Models: A Surveycited by 221
  3. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  6. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  7. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  8. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
  9. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 56
  10. 2026MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Serverscited by 46
  11. 2024Benchmarking Agentic Workflow Generationcited by 36
  12. 2025RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solvingcited by 19
23 more
  1. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environmentscited by 19
  2. 2024ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMscited by 17
  3. 2026ToolGate: Contract-Grounded and Verified Tool Execution for LLMscited by 13
  4. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  5. 2025GTM: Simulating the World of Tools for AI Agentscited by 12
  6. 2024TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Usecited by 11
  7. 2025TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Taskscited by 9
  8. 2026CL-bench Life: Can Language Models Learn from Real-Life Context?cited by 8
  9. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6
  10. 2025GenoMAS: A Multi-Agent Framework for Scientific Discovery via Code-Driven Gene Expression Analysiscited by 6
  11. 2026When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agentscited by 5
  12. 2024NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of Large Language Modelscited by 4
  13. 2026AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarioscited by 3
  14. 2026Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Usecited by 3
  15. 202460 Data Points are Sufficient to Fine-Tune LLMs for Question-Answeringcited by 2
  16. 2025LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Modelscited by 2
  17. 2026NC2C: Automated Convexification of Generic Non-Convex Optimization Problemscited by 2
  18. 2026Concurrency without Model Changes: Future-based Asynchronous Function Calling for LLMscited by 2
  19. 2025Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistantcited by 1
  20. 2025Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humanscited by 1
  21. 2026MetaToolAgent: Towards Generalizable Tool Usage in LLMs through Meta-Learningcited by 1
  22. 2026NTILC: Neural Tool Invocation via Learned Compressioncited by 1
  23. 2025Re-Initialization Token Learning for Tool-Augmented Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
  2. 2023Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model
  3. 2024LLM With Tools: A Survey
  4. 2025ToLeaP: Rethinking Development of Tool Learning with Large Language Models
  5. 2024What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks
  6. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning
  7. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Models
  8. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios