mlsanityData to 7 Oct 2026

API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs

arXiv 2402.15491 ↗First posted 23 Feb 2024cs.CL cs.AIcited by 39

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

There is a growing need for Large Language Models (LLMs) to effectively use tools and external Application Programming Interfaces (APIs) to plan and complete tasks. As such, there is tremendous interest in methods that can acquire sufficient quantities of train and test data that involve calls to tools / APIs. Two lines of research have emerged as the predominant strategies for addressing this challenge. The first has focused on synthetic data generation techniques, while the second has involved curating task-adjacent datasets which can be transformed into API / Tool-based tasks. In this paper, we focus on the task of identifying, curating, and transforming existing datasets and, in turn, introduce API-BLEND, a large corpora for training and systematic testing of tool-augmented LLMs. The datasets mimic real-world scenarios involving API-tasks such as API / tool detection, slot filling, and sequencing of the detected APIs. We demonstrate the utility of the API-BLEND dataset for both training and benchmarking purposes.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 31 of its 42 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

31 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  3. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  4. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  5. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  6. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  12. 2023StarCoder: may the source be with you!cited by 790
12 more
  1. 2023The Falcon Series of Open Language Modelscited by 526
  2. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  4. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  5. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  6. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  7. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  8. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  9. 2021MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluationcited by 86
  10. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
  11. 2023ToolTalk: Evaluating Tool-Usage in a Conversational Settingcited by 42
  12. 2019Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT qualitycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 20; 2026: 7.

39 papers.

Sort:

By area:

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  2. 2024Tool Learning with Large Language Models: A Surveycited by 221
  3. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  4. 2025Survey on Evaluation of LLM-based Agentscited by 190
  5. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 116
  6. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 61
  7. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Maskingcited by 53
  8. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  9. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Callscited by 45
  10. 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 44
  11. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 37
  12. 2025When2Call: When (not) to Call Toolscited by 36
27 more
  1. 2024ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesiscited by 27
  2. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  3. 2024On Mitigating Code LLM Hallucinations with API Documentationcited by 21
  4. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  5. 2024MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domainscited by 18
  6. 2025Tool Unlearning for Tool-Augmented LLMscited by 17
  7. 2025StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIscited by 14
  8. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  9. 2025GTM: Simulating the World of Tools for AI Agentscited by 12
  10. 2024Scaling Granite Code Models to 128K Contextcited by 9
  11. 2025Chain-of-Tools: Utilizing Massive Unseen Tools in the CoT Reasoning of Frozen Language Modelscited by 9
  12. 2025ToolForge: A Data Synthesis Pipeline for Multi-Hop Search without Real-World APIscited by 8
  13. 2025Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Callingcited by 7
  14. 2025ProAgent: Harnessing On-Demand Sensory Contexts for Proactive LLM Agent Systems in the Wildcited by 7
  15. 2025FamilyTool: A Multi-hop Personalized Tool Use Benchmarkcited by 5
  16. 2025Delegated Authorization for Agents Constrained to Semantic Task-to-Scope Matchingcited by 5
  17. 2026Investigating Tool-Memory Conflicts in Tool-Augmented LLMscited by 5
  18. 2026Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learningcited by 5
  19. 2024NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of Large Language Modelscited by 4
  20. 2025Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Callingcited by 4
  21. 2025An Auditable Agent Platform For Automated Molecular Optimisationcited by 2
  22. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  23. 2026ToolWeave: Structured Synthesis of Complex Multi-Turn Tool-Calling Dialoguescited by 1
  24. 2026Continual Model Routing in Evolving Model Hubscited by 1
  25. 2026GenesisFunc: Multi-Agent Data Generation for Accurate and Generalizable Function-Callingcited by 1
  26. 2026The Bitter Lesson of Tool Callingcited by 1
  27. 2026Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation
  2. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
  3. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
  4. 2025Evaluating LLMs on Sequential API Call Through Automated Test Generation
  5. 2026Enhancing Tool Calling in LLMs with the International Tool Calling Dataset
  6. 2024Octopus: On-device language model for function calling of software APIs
  7. 2025In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents
  8. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models