mlsanityData to 7 Oct 2026

ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

arXiv 2307.16789 ↗First posted 31 Jul 2023cs.AI cs.CL cs.LGcited by 1,880

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Despite the advancements of open-source large language models (LLMs), e.g., LLaMA, they remain significantly limited in tool-use capabilities, i.e., using external tools (APIs) to fulfill human instructions. The reason is that current instruction tuning largely focuses on basic language tasks but ignores the tool-use domain. This is in contrast to the excellent tool-use capabilities of state-of-the-art (SOTA) closed-source LLMs, e.g., ChatGPT. To bridge this gap, we introduce ToolLLM, a general tool-use framework encompassing data construction, model training, and evaluation. We first present ToolBench, an instruction-tuning dataset for tool use, which is constructed automatically using ChatGPT. Specifically, the construction can be divided into three stages: (i) API collection: we collect 16,464 real-world RESTful APIs spanning 49 categories from RapidAPI Hub; (ii) instruction generation: we prompt ChatGPT to generate diverse instructions involving these APIs, covering both single-tool and multi-tool scenarios; (iii) solution path annotation: we use ChatGPT to search for a valid solution path (chain of API calls) for each instruction. To enhance the reasoning capabilities of LLMs, we develop a novel depth-first search-based decision tree algorithm. It enables LLMs to evaluate multiple reasoning traces and expand the search space. Moreover, to evaluate the tool-use capabilities of LLMs, we develop an automatic evaluator: ToolEval. Based on ToolBench, we fine-tune LLaMA to obtain an LLM ToolLLaMA, and equip it with a neural API retriever to recommend appropriate APIs for each instruction. Experiments show that ToolLLaMA demonstrates a remarkable ability to execute complex instructions and generalize to unseen APIs, and exhibits comparable performance to ChatGPT. Our ToolLLaMA also demonstrates strong zero-shot generalization ability in an out-of-distribution tool-use dataset: APIBench.

Key facts. Cited by 1,880 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 40 of its 65 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

40 of 65 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  3. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  4. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  5. 2023Tool Learning with Foundation Models17 Apr
  6. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
21 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  11. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  12. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  13. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  14. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  15. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  16. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  17. 2023GeneGPT: Augmenting Large Language Models with Domain Tools for Improved Access to Biomedical Informationcited by 115
  18. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
  19. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  20. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90
  21. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 88; 2024: 347; 2025: 468; 2026: 977.

1,880 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  5. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  6. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  7. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  8. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  9. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  10. 2025Kimi K2: Open Agentic Intelligencecited by 523
  11. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  12. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
38 more
  1. 2023Cognitive Architectures for Language Agentscited by 422
  2. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  3. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  4. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  5. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  6. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  7. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  8. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  9. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  11. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  12. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  13. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  14. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  15. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  16. 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
  17. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 268
  18. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  19. 2025Defeating Prompt Injections by Designcited by 256
  20. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  21. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  22. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  23. 2024Capabilities of Gemini Models in Medicinecited by 231
  24. 2024Tool Learning with Large Language Models: A Surveycited by 221
  25. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  26. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  27. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  28. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  29. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  30. 2025Survey on Evaluation of LLM-based Agentscited by 190
  31. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  32. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  33. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  34. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  35. 2023An LLM Compiler for Parallel Function Callingcited by 170
  36. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 169
  37. 2025Agentic Reinforced Policy Optimizationcited by 166
  38. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160

50 most cited of 1,880 All 1,880 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation
  2. 2024CodecLM: Aligning Language Models with Tailored Synthetic Data
  3. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  4. 2024Language hooks: a modular framework for augmenting LLM reasoning that decouples tool usage from the model and its prompt
  5. 2025ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models
  6. 2025StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs
  7. 2024MAmmoTH2: Scaling Instructions from the Web
  8. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermism