# TaskBench: Benchmarking Large Language Models for Task Automation

arXiv: https://arxiv.org/abs/2311.18760 · HTML: https://mlsanity.com/paper/2311.18760/

## Key facts

- Cited by 144 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs](https://mlsanity.com/paper/2303.16434/) (2023)

## Built on by (most cited first, 30 of 144)

- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [From Persona to Personalization: A Survey on Role-Playing Language Agents](https://mlsanity.com/paper/2404.18231/) (2024)
- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction](https://mlsanity.com/paper/2401.06201/) (2024)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms](https://mlsanity.com/paper/2406.14228/) (2024)
- [Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization](https://mlsanity.com/paper/2402.17574/) (2024)
- [Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale](https://mlsanity.com/paper/2603.02176/) (2026)
- [MindSearch: Mimicking Human Minds Elicits Deep AI Searcher](https://mlsanity.com/paper/2407.20183/) (2024)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [LLM With Tools: A Survey](https://mlsanity.com/paper/2409.18807/) (2024)
- [What Are Tools Anyway? A Survey from the Language Model Perspective](https://mlsanity.com/paper/2403.15452/) (2024)
- [SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories](https://mlsanity.com/paper/2409.07440/) (2024)
- [Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usage](https://mlsanity.com/paper/2412.15606/) (2024)
- [ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use](https://mlsanity.com/paper/2501.02506/) (2025)
- [Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models](https://mlsanity.com/paper/2503.01763/) (2025)
- [Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks](https://mlsanity.com/paper/2503.21696/) (2025)
- [Can Graph Learning Improve Planning in LLM-based Agents?](https://mlsanity.com/paper/2405.19119/) (2024)
- [Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning](https://mlsanity.com/paper/2410.12952/) (2024)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks](https://mlsanity.com/paper/2403.11085/) (2024)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Benchmarking Agentic Workflow Generation](https://mlsanity.com/paper/2410.07869/) (2024)
- [ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios](https://mlsanity.com/paper/2401.00741/) (2024)
- [LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations](https://mlsanity.com/paper/2412.01441/) (2024)
- [AI PERSONA: Towards Life-long Personalization of LLMs](https://mlsanity.com/paper/2412.13103/) (2024)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- [Advancing Agentic Systems: Dynamic Task Decomposition, Tool Integration and Evaluation using Novel Metrics and Dataset](https://mlsanity.com/paper/2410.22457/) (2024)

## Abstract (arXiv)

In recent years, the remarkable progress of large language models (LLMs) has sparked interest in task automation, which involves decomposing complex tasks described by user instructions into sub-tasks and invoking external tools to execute them, playing a central role in autonomous agents. However, there is a lack of systematic and standardized benchmarks to promote the development of LLMs in task automation. To address this, we introduce TaskBench, a comprehensive framework to evaluate the capability of LLMs in task automation. Specifically, task automation can be divided into three critical stages: task decomposition, tool selection, and parameter prediction. To tackle the complexities inherent in these stages, we introduce the concept of Tool Graph to represent decomposed tasks and adopt a back-instruct method to generate high-quality user instructions. We propose TaskEval, a multi-faceted evaluation methodology that assesses LLM performance across these three stages. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation. Experimental results demonstrate that TaskBench effectively reflects the capabilities of various LLMs in task automation. It provides insights into model performance across different task complexities and domains, pushing the boundaries of what current models can achieve. TaskBench offers a scalable, adaptable, and reliable benchmark for advancing LLM-based autonomous agents.