mlsanityData to 7 Oct 2026

T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step

arXiv 2312.14033 ↗First posted 21 Dec 2023cs.CLcited by 91

Not on the map. Why?

Cited by 65 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLM) have achieved remarkable performance on various NLP tasks and are augmented by tools for broader applications. Yet, how to evaluate and analyze the tool-utilization capability of LLMs is still under-explored. In contrast to previous works that evaluate models holistically, we comprehensively decompose the tool utilization into multiple sub-processes, including instruction following, planning, reasoning, retrieval, understanding, and review. Based on that, we further introduce T-Eval to evaluate the tool utilization capability step by step. T-Eval disentangles the tool utilization evaluation into several sub-domains along model capabilities, facilitating the inner understanding of both holistic and isolated competency of LLMs. We conduct extensive experiments on T-Eval and in-depth analysis of various LLMs. T-Eval not only exhibits consistency with the outcome-oriented evaluation but also provides a more fine-grained analysis of the capabilities of LLMs, providing a new perspective in LLM evaluation on tool-utilization ability. The benchmark will be available at https://github.com/open-compass/T-Eval.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

31 of 53 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
19 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  11. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  12. 2023Augmented Language Models: a Surveycited by 398
  13. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  14. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  15. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  16. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  17. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  18. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  19. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 39; 2026: 38.

91 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024Tool Learning with Large Language Models: A Surveycited by 221
  3. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  4. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  5. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  6. 2025LocAgent: Graph-Guided LLM Agents for Code Localizationcited by 82
  7. 2024Attention Heads of Large Language Models: A Surveycited by 65
  8. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 56
  9. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 48
  10. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  11. 2024FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agentscited by 39
  12. 2024Benchmarking Agentic Workflow Generationcited by 36
38 more
  1. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  2. 2025GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modelingcited by 35
  3. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 31
  4. 2024CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Masterycited by 30
  5. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agentscited by 22
  6. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  7. 2025RobustFlow: Towards Robust Agentic Workflow Generationcited by 19
  8. 2024Enhancing Tool Retrieval with Iterative Feedback from Large Language Modelscited by 17
  9. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 15
  10. 2026Benchmarking LLM Tool-Use in the Wildcited by 15
  11. 2025StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIscited by 14
  12. 2025From REST to MCP: An Empirical Study of API Wrapping and Automated Server Generation for LLM Agentscited by 14
  13. 2025Generalizable End-to-End Tool-Use RL with Synthetic CodeGymcited by 14
  14. 2025SimuHome: A Temporal- and Environment-Aware Benchmark for Smart Home LLM Agentscited by 13
  15. 2025MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environmentscited by 13
  16. 2024MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verificationcited by 12
  17. 2024TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Usecited by 11
  18. 2026Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Modelscited by 10
  19. 2025Benchmarking Failures in Tool-Augmented Language Modelscited by 9
  20. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missionscited by 9
  21. 2025A Framework for Testing and Adapting REST APIs as LLM Toolscited by 9
  22. 2025TPS-Bench: Evaluating AI Agents' Tool Planning & Scheduling Abilities in Compounding Taskscited by 9
  23. 2026EnterpriseBench Corecraft: Training Generalizable Agents on High-Fidelity RL Environmentscited by 8
  24. 2025ToolACE-DEV: Self-Improving Tool Learning via Decomposition and EVolutioncited by 7
  25. 2026ACE-Router: Generalizing History-Aware Routing from MCP Tools to the Agent Webcited by 7
  26. 2026PrefIx: Understand and Adapt to User Preference in Human-Agent Interactioncited by 7
  27. 2026CCTU: A Benchmark for Tool Use under Complex Constraintscited by 7
  28. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6
  29. 2025ToolSpectrum : Towards Personalized Tool Utilization for Large Language Modelscited by 6
  30. 2025DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialoguescited by 6
  31. 2025Don't Adapt Small Language Models for Tools; Adapt Tool Schemas to the Modelscited by 6
  32. 2025FamilyTool: A Multi-hop Personalized Tool Use Benchmarkcited by 5
  33. 2025ToolHaystack: Stress-Testing Tool-Augmented Language Models in Realistic Long-Term Interactionscited by 5
  34. 2025EHR-MCP: Real-world Evaluation of Clinical Information Retrieval by Large Language Models via Model Context Protocolcited by 5
  35. 2026Beyond Function Calling: Benchmarking Tool-Using Agents under Tool-Environment Unreliabilitycited by 5
  36. 2024Mixture-of-Instructions: Aligning Large Language Models via Mixture Promptingcited by 4
  37. 2024NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of Large Language Modelscited by 4
  38. 2025TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devicescited by 4

50 most cited of 91 All 91 (JSON)