mlsanityData to 7 Oct 2026

Tool Learning with Large Language Models: A Survey

arXiv 2405.17935 ↗First posted 28 May 2024cs.CL cs.AIcited by 221

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Recently, tool learning with large language models (LLMs) has emerged as a promising paradigm for augmenting the capabilities of LLMs to tackle highly complex problems. Despite growing attention and rapid advancements in this field, the existing literature remains fragmented and lacks systematic organization, posing barriers to entry for newcomers. This gap motivates us to conduct a comprehensive survey of existing works on tool learning with LLMs. In this survey, we focus on reviewing existing literature from the two primary aspects (1) why tool learning is beneficial and (2) how tool learning is implemented, enabling a comprehensive understanding of tool learning with LLMs. We first explore the "why" by reviewing both the benefits of tool integration and the inherent benefits of the tool learning paradigm from six specific aspects. In terms of "how", we systematically review the literature according to a taxonomy of four key stages in the tool learning workflow: task planning, tool selection, tool calling, and response generation. Additionally, we provide a detailed summary of existing benchmarks and evaluation methods, categorizing them according to their relevance to different stages. Finally, we discuss current challenges and outline potential future directions, aiming to inspire both researchers and industrial developers to further explore this emerging and promising area. We also maintain a GitHub repository to continually keep track of the relevant papers and resources in this rising area at https://github.com/quchangle1/LLM-Tool-Survey.

Key facts. Cited by 221 papers in the map (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 160 of its 215 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

160 of 215 references.

Closest ancestors · 33

Same direction, newest first

  1. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents26 May
  2. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark14 May
  3. 2024Planning and Editing What You Retrieve for Enhanced Tool Learning30 Mar
  4. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models12 Mar
  5. 2024LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error7 Mar
  6. 2024Learning to Use Tools via Cooperative and Interactive Agents5 Mar
  7. 2024Budget-Constrained Tool Learning with Planning25 Feb
  8. 2024API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs23 Feb
  9. 2024TOOLVERIFIER: Generalization to New Tools via Self-Verification21 Feb
  10. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages16 Feb
  11. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls6 Feb
  12. 2024Efficient Tool Use with Chain-of-Abstraction Reasoning30 Jan
21 more
  1. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning16 Jan
  2. 2024EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction11 Jan
  3. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios1 Jan
  4. 2023Reverse Chain: A Generic-Rule for LLMs to Master Multi-API Planning6 Oct
  5. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  6. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback19 Sep
  7. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum27 Aug
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  9. 2023GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution17 Jul
  10. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  11. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  12. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  13. 2023Large Language Models as Tool Makers26 May
  14. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  15. 2023Making Language Models Better Tool Learners with Execution Feedback22 May
  16. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  17. 2023Tool Learning with Foundation Models17 Apr
  18. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr
  19. 2023ART: Automatic multi-step reasoning and tool-use for large language models16 Mar
  20. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb
  21. 2022TALM: Tool Augmented Language Models24 May

Other references · 127

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2021Program Synthesis with Large Language Modelscited by 3,509
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  12. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
38 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  5. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  7. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  13. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  14. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  15. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  16. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  17. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  18. 2022PAL: Program-aided Language Modelscited by 635
  19. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  20. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  21. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  22. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  23. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  24. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  25. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  26. 2023Cognitive Architectures for Language Agentscited by 422
  27. 2024Understanding the planning of LLM agents: A surveycited by 414
  28. 2023Augmented Language Models: a Surveycited by 398
  29. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  30. 2023Language Models can Solve Computer Taskscited by 371
  31. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  32. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  33. 2023Explainability for Large Language Models: A Surveycited by 331
  34. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  35. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  36. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  37. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  38. 2022Reasoning with Language Model Prompting: A Surveycited by 251

50 most cited of 127 All 127 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 107; 2026: 74.

221 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  4. 2024Automated Design of Agentic Systemscited by 360
  5. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  6. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  7. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
  8. 2025Deep Research Agents: A Systematic Examination And Roadmapcited by 122
  9. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 105
  10. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  11. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  12. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
38 more
  1. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 86
  2. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  3. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 82
  4. 2025A Survey of AI Agent Protocolscited by 77
  5. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 72
  6. 2026Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineeringcited by 68
  7. 2025ACEBench: Who Wins the Match Point in Tool Usage?cited by 66
  8. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 66
  9. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 64
  10. 2024Direct Multi-Turn Preference Optimization for Language Agentscited by 63
  11. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  12. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  13. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  14. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  15. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  16. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 48
  17. 2025LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?cited by 47
  18. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  19. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
  20. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 46
  21. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 43
  22. 2025Step-GUI Technical Reportcited by 38
  23. 2024Benchmarking Agentic Workflow Generationcited by 36
  24. 2025When2Call: When (not) to Call Toolscited by 36
  25. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  26. 2026When Single-Agent with Skills Replace Multi-Agent Systems and When They Failcited by 34
  27. 2026CUA-Skill: Develop Skills for Computer Using Agentcited by 32
  28. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 31
  29. 2024LAMBDA: A Large Model Based Data Agentcited by 30
  30. 2025Adaptive Tool Use in Large Language Models with Meta-Cognition Triggercited by 29
  31. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvementcited by 27
  32. 2025Taming Knowledge Conflicts in Language Modelscited by 27
  33. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  34. 2024AppBench: Planning of Multiple APIs from Various APPs for Complex User Instructioncited by 24
  35. 2025DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Taskscited by 24
  36. 2024Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Basescited by 23
  37. 2024Aviary: training language agents on challenging scientific taskscited by 23
  38. 2024Kolb-Based Experiential Learning for Generalist Agents with Human-Level Kaggle Data Science Performancecited by 22

50 most cited of 221 All 221 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM With Tools: A Survey
  2. 2025ToLeaP: Rethinking Development of Tool Learning with Large Language Models
  3. 2024What Affects the Stability of Tool Learning? An Empirical Study on the Robustness of Tool Learning Frameworks
  4. 2024Tool-Planner: Task Planning with Clusters across Multiple Tools
  5. 2026Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction
  6. 2024WTU-EVAL: A Whether-or-Not Tool Usage Evaluation Benchmark for Large Language Models
  7. 2024Towards Practical Tool Usage for Continually Learning LLMs
  8. 2025Alignment for Efficient Tool Calling of Large Language Models