mlsanityData to 7 Oct 2026

APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets

arXiv 2406.18518 ↗First posted 26 Jun 2024cs.CL cs.AI cs.LGcited by 188

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

The advancement of function-calling agent models requires diverse, reliable, and high-quality datasets. This paper presents APIGen, an automated data generation pipeline designed to synthesize verifiable high-quality datasets for function-calling applications. We leverage APIGen and collect 3,673 executable APIs across 21 different categories to generate diverse function-calling datasets in a scalable and structured manner. Each data in our dataset is verified through three hierarchical stages: format checking, actual function executions, and semantic verification, ensuring its reliability and correctness. We demonstrate that models trained with our curated datasets, even with only 7B parameters, can achieve state-of-the-art performance on the Berkeley Function-Calling Benchmark, outperforming multiple GPT-4 models. Moreover, our 1B model achieves exceptional performance, surpassing GPT-3.5-Turbo and Claude-3 Haiku. We release a dataset containing 60,000 high-quality entries, aiming to advance the field of function-calling agent domains. The dataset is available on Huggingface: https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k and the project homepage: https://apigen-pipeline.github.io/

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 37 of its 54 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

37 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models12 Mar
  2. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  3. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback19 Sep
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  5. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  6. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  7. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  7. 2024Mixtral of Expertscited by 1,785
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  10. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  11. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
17 more
  1. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  5. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  6. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  7. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  8. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  9. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  10. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 83
  11. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  12. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
  13. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  14. 2024AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent Systemcited by 53
  15. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  16. 2023Creative Robot Tool Use with Large Language Modelscited by 42
  17. 2024Octopus v4: Graph of language modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 80; 2026: 88.

188 papers.

Sort:

By area:

  1. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  2. 2025Survey on Evaluation of LLM-based Agentscited by 190
  3. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 167
  4. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  5. 2024xLAM: A Family of Large Action Models to Empower AI Agent Systemscited by 108
  6. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  7. 2025ACEBench: Who Wins the Match Point in Tool Usage?cited by 66
  8. 2024ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learningcited by 61
  9. 2024Hammer: Robust Function-Calling for On-Device Language Models via Function Maskingcited by 53
  10. 2025Simulating Environments with Reasoning Models for Agent Trainingcited by 52
  11. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  12. 2025Towards General Agentic Intelligence via Environment Scalingcited by 51
38 more
  1. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 51
  2. 2024Multi-modal Agent Tuning: Building a VLM-Driven Agent for Efficient Tool Usagecited by 49
  3. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  4. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Callscited by 45
  5. 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 40
  6. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  7. 2025JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agentcited by 39
  8. 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
  9. 2025When2Call: When (not) to Call Toolscited by 36
  10. 2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 29
  11. 2025SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentscited by 29
  12. 2024ToolFlow: Boosting LLM Tool-Calling Through Natural and Coherent Dialogue Synthesiscited by 27
  13. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 23
  14. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learningcited by 22
  15. 2025Procedural Environment Generation for Tool-Use Agentscited by 21
  16. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  17. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Modelscited by 19
  18. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  19. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agentscited by 17
  20. 2025On the Robustness of Agentic Function Callingcited by 17
  21. 2025LongFuncEval: Measuring the effectiveness of long context models for function callingcited by 17
  22. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratchcited by 17
  23. 2026From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agentscited by 17
  24. 2025LoopTool: Closing the Data-Training Loop for Robust LLM Tool Callscited by 16
  25. 2025CoRT: Code-integrated Reasoning within Thinkingcited by 15
  26. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  27. 2026Benchmarking LLM Tool-Use in the Wildcited by 15
  28. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 15
  29. 2024Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translationcited by 14
  30. 2025Generalizable End-to-End Tool-Use RL with Synthetic CodeGymcited by 14
  31. 2026Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Textcited by 13
  32. 2026Terminal-World: Scaling Terminal-Agent Environments via Agent Skillscited by 13
  33. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  34. 2025AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarificationcited by 12
  35. 2025MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agentscited by 12
  36. 2025World Modelling Improves Language Model Agentscited by 12
  37. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  38. 2025GTM: Simulating the World of Tools for AI Agentscited by 12

50 most cited of 188 All 188 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GenesisFunc: Multi-Agent Data Generation for Accurate and Generalizable Function-Calling
  2. 2026Simulate to Generalize: Scaling Stateful Supervision for API-calling Agents using LLM World Models
  3. 2025FunReason-MT Technical Report: Advanced Data Synthesis Solution for Real-world Multi-Turn Tool-use
  4. 2026Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs
  5. 2026Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity
  6. 2025CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
  7. 2025StackPilot: Autonomous Function Agents for Scalable and Environment-Free Code Execution
  8. 2026AgentWeave: Routing Before Reasoning for Efficient Function Calling in Tool-Rich Language Models