mlsanityData to 7 Oct 2026

API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs

arXiv 2304.08244 ↗First posted 14 Apr 2023cs.CL cs.AIcited by 574

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. However, three pivotal questions remain unanswered: (1) How effective are current LLMs in utilizing tools? (2) How can we enhance LLMs' ability to utilize tools? (3) What obstacles need to be overcome to leverage tools? To address these questions, we introduce API-Bank, a groundbreaking benchmark, specifically designed for tool-augmented LLMs. For the first question, we develop a runnable evaluation system consisting of 73 API tools. We annotate 314 tool-use dialogues with 753 API calls to assess the existing LLMs' capabilities in planning, retrieving, and calling APIs. For the second question, we construct a comprehensive training set containing 1,888 tool-use dialogues from 2,138 APIs spanning 1,000 distinct domains. Using this dataset, we train Lynx, a tool-augmented LLM initialized from Alpaca. Experimental results demonstrate that GPT-3.5 exhibits improved tool utilization compared to GPT-3, while GPT-4 excels in planning. However, there is still significant potential for further improvement. Moreover, Lynx surpasses Alpaca's tool utilization performance by more than 26 pts and approaches the effectiveness of GPT-3.5. Through error analysis, we highlight the key challenges for future research in this field to answer the third question.

Key facts. Cited by 574 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 13 of its 28 references that are in the map. First posted on arXiv 14 Apr 2023.

Builds on

13 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  10. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  11. 2023Augmented Language Models: a Surveycited by 398
  12. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 85; 2025: 147; 2026: 324.

574 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  3. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  4. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  5. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  6. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  7. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  8. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  9. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  10. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  11. 2024Tool Learning with Large Language Models: A Surveycited by 221
  12. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
38 more
  1. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  2. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  3. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instructioncited by 153
  6. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  7. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  8. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  9. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  10. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 129
  11. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  12. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  13. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
  14. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  15. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  16. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  17. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  18. 2026AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolutioncited by 105
  19. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  20. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  21. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  22. 2025AgentNet: Decentralized Evolutionary Coordination for LLM-based Multi-Agent Systemscited by 86
  23. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 85
  24. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 80
  25. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  26. 2024GTA: A Benchmark for General Tool Agentscited by 77
  27. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 75
  28. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  29. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  30. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  31. 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 69
  32. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 67
  33. 2024MindSearch: Mimicking Human Minds Elicits Deep AI Searchercited by 65
  34. 2026Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scalecited by 65
  35. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmarkcited by 62
  36. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  37. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  38. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53

50 most cited of 574 All 574 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation
  2. 2026Enhancing Tool Calling in LLMs with the International Tool Calling Dataset
  3. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs
  4. 2024LLM With Tools: A Survey
  5. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
  6. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
  7. 2025OR-Toolformer: Modeling and Solving Operations Research Problems with Tool Augmented Large Language Models
  8. 2023Gorilla: Large Language Model Connected with Massive APIs