mlsanityData to 14 Oct 2026

TaskBench: Benchmarking Large Language Models for Task Automation

arXiv 2311.18760 ↗First posted 30 Nov 2023cs.CL cs.AIcited by 144

Not on the map. Why?

Cited by 79 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In recent years, the remarkable progress of large language models (LLMs) has sparked interest in task automation, which involves decomposing complex tasks described by user instructions into sub-tasks and invoking external tools to execute them, playing a central role in autonomous agents. However, there is a lack of systematic and standardized benchmarks to promote the development of LLMs in task automation. To address this, we introduce TaskBench, a comprehensive framework to evaluate the capability of LLMs in task automation. Specifically, task automation can be divided into three critical stages: task decomposition, tool selection, and parameter prediction. To tackle the complexities inherent in these stages, we introduce the concept of Tool Graph to represent decomposed tasks and adopt a back-instruct method to generate high-quality user instructions. We propose TaskEval, a multi-faceted evaluation methodology that assesses LLM performance across these three stages. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation. Experimental results demonstrate that TaskBench effectively reflects the capabilities of various LLMs in task automation. It provides insights into model performance across different task complexities and domains, pushing the boundaries of what current models can achieve. TaskBench offers a scalable, adaptable, and reliable benchmark for advancing LLM-based autonomous agents.

Key facts. Cited by 144 papers in the map (data through 14 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

28 of 46 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
16 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  11. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  12. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  13. 2023Tool Learning with Foundation Modelscited by 302
  14. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  15. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  16. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 36; 2025: 39; 2026: 68.

144 papers.

Sort:

By area:

  1. 2024Tool Learning with Large Language Models: A Surveycited by 223
  2. 2024AI Agents That Mattercited by 219
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  4. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 172
  5. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 154
  6. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  7. 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithmscited by 113
  8. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 94
  9. 2026Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scalecited by 72
  10. 2024MindSearch: Mimicking Human Minds Elicits Deep AI Searchercited by 66
  11. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMscited by 64
  12. 2024LLM With Tools: A Surveycited by 61
38 more
  1. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 54
  2. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  3. 2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 51
  4. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 50
  5. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 47
  6. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 44
  7. 2024Can Graph Learning Improve Planning in LLM-based Agents?cited by 43
  8. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 42
  9. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  10. 2024m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Taskscited by 38
  11. 2024Foundation Models for Music: A Surveycited by 37
  12. 2024Benchmarking Agentic Workflow Generationcited by 37
  13. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  14. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  15. 2024AI PERSONA: Towards Life-long Personalization of LLMscited by 32
  16. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  17. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  18. 2024Advancing Agentic Systems: Dynamic Task Decomposition, Tool Integration and Evaluation using Novel Metrics and Datasetcited by 24
  19. 2024TimeArena: Shaping Efficient Multitasking Language Agents in a Time-Aware Simulationcited by 23
  20. 2024Adapting LLMs for Efficient Context Processing through Soft Prompt Compressioncited by 22
  21. 2025Latent Inter-User Difference Modeling for LLM Personalizationcited by 21
  22. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 20
  23. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 19
  24. 2025CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarioscited by 19
  25. 2025ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filteringcited by 19
  26. 2026SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfacescited by 19
  27. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applicationscited by 18
  28. 2026Benchmarking LLM Tool-Use in the Wildcited by 17
  29. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
  30. 2026Solver-Aided Verification of Policy Compliance in Tool-Augmented LLM Agentscited by 16
  31. 2024BPP-Search: Enhancing Tree of Thought Reasoning for Mathematical Modeling Problem Solvingcited by 15
  32. 2025AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenancecited by 15
  33. 2026Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistantscited by 15
  34. 2024CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planningcited by 14
  35. 2025PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Playcited by 14
  36. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  37. 2025AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarificationcited by 13
  38. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 13

50 most cited of 144 All 144 (JSON)