# ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents

arXiv: https://arxiv.org/abs/2407.00132 · HTML: https://mlsanity.com/paper/2407.00132/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 21 of 21)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [On the Tool Manipulation Capability of Open-source Large Language Models](https://mlsanity.com/paper/2305.16504/) (2023)
- [Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents](https://mlsanity.com/paper/2402.09205/) (2024)
- [KwaiAgents: Generalized Information-seeking Agent System with Large Language Models](https://mlsanity.com/paper/2312.04889/) (2023)
- [From Robotic Process Automation to Intelligent Process Automation: Emerging Trends](https://mlsanity.com/paper/2007.13257/) (2020)
- [IFTTT vs. Zapier: A Comparative Study of Trigger-Action Programming Frameworks](https://mlsanity.com/paper/1709.02788/) (2017)

## Built on by (most cited first, 30 of 36)

- [Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents](https://mlsanity.com/paper/2503.24047/) (2025)
- [Beyond Browsing: API-Based Web Agents](https://mlsanity.com/paper/2410.16464/) (2024)
- [OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows](https://mlsanity.com/paper/2508.09124/) (2025)
- [NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls](https://mlsanity.com/paper/2409.03797/) (2024)
- [LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?](https://mlsanity.com/paper/2508.01780/) (2025)
- [Learning to Ask: When LLM Agents Meet Unclear Instruction](https://mlsanity.com/paper/2409.00557/) (2024)
- [MagCache: Fast Video Generation with Magnitude-Aware Cache](https://mlsanity.com/paper/2506.09045/) (2025)
- [WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models](https://mlsanity.com/paper/2411.05451/) (2024)
- [Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMs](https://mlsanity.com/paper/2502.07942/) (2025)
- [AI Scientists Fail Without Strong Implementation Capability](https://mlsanity.com/paper/2506.01372/) (2025)
- [WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seeking](https://mlsanity.com/paper/2510.24697/) (2025)
- [MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2602.19843/) (2026)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs](https://mlsanity.com/paper/2503.20527/) (2025)
- [Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk](https://mlsanity.com/paper/2502.15865/) (2025)
- [Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missions](https://mlsanity.com/paper/2504.02623/) (2025)
- [AD-AGENT: A Multi-agent Framework for End-to-end Anomaly Detection](https://mlsanity.com/paper/2505.12594/) (2025)
- [SkillRAE: Agent Skill-Based Context Compilation for Retrieval-Augmented Execution](https://mlsanity.com/paper/2605.10114/) (2026)
- [DroidCall: A Dataset for LLM-powered Android Intent Invocation](https://mlsanity.com/paper/2412.00402/) (2024)
- [FamilyTool: A Multi-hop Personalized Tool Use Benchmark](https://mlsanity.com/paper/2504.06766/) (2025)
- [DRAGON: Domain-specific Robust Automatic Data Generation for RAG Optimization](https://mlsanity.com/paper/2505.10989/) (2025)
- [C3-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking](https://mlsanity.com/paper/2505.18746/) (2025)
- [Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent](https://mlsanity.com/paper/2603.05578/) (2026)
- [PEToolLLM: Towards Personalized Tool Learning in Large Language Models](https://mlsanity.com/paper/2502.18980/) (2025)
- [OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction](https://mlsanity.com/paper/2602.15197/) (2026)
- [Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language](https://mlsanity.com/paper/2604.19667/) (2026)
- [Evaluating LLMs on Sequential API Call Through Automated Test Generation](https://mlsanity.com/paper/2507.09481/) (2025)
- [In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents](https://mlsanity.com/paper/2509.01560/) (2025)
- [Beyond Static Endpoints: Tool Programs as an Interface for Flexible Agentic Web Services](https://mlsanity.com/paper/2606.19992/) (2026)
- [Blockchain Empowered Trustworthy Agent Networks: Foundations, Taxonomy, and Future Directions](https://mlsanity.com/paper/2608.04626/) (2026)

## Abstract (arXiv)

Recent advancements in integrating large language models (LLMs) with application programming interfaces (APIs) have gained significant interest in both academia and industry. Recent work demonstrates that these API-based agents exhibit relatively strong autonomy and planning capabilities. However, their ability to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unknown. In this paper, we introduce textscShortcutsBench, a large-scale benchmark for the comprehensive evaluation of API-based agents in solving real-world complex tasks. textscShortcutsBench includes a wealth of real APIs from Apple Inc., refined user queries, human-annotated high-quality action sequences, detailed parameter filling values, and parameters requesting necessary input from the system or user. We revealed how existing benchmarks~/~datasets struggle to accommodate the advanced reasoning capabilities of existing more intelligent LLMs. Moreover, our extensive evaluation of agents built with 5 leading open-source (size ≥ 57B) and 5 closed-source LLMs (e.g. Gemini-1.5-Pro and GPT-4o-mini) with varying intelligence level reveals significant limitations of existing API-based agents in the whole process of handling complex queries related to API selection, parameter filling, and requesting necessary input from the system and the user. These findings highlight the great challenges that API-based agents face in effectively fulfilling real and complex user queries. All datasets, code, experimental logs, and results are available at urlhttps://github.com/EachSheep/ShortcutsBench.