mlsanityData to 14 Oct 2026

LLM With Tools: A Survey

arXiv 2409.18807 ↗First posted 24 Sep 2024cs.AIcited by 61

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

The integration of tools in augmenting large language models presents a novel approach toward enhancing the efficiency and accuracy of these models in handling specific, complex tasks. This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools, thereby pushing the boundaries of their capabilities beyond pre-existing knowledge bases. We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans and their execution, emphasizing the significance of understanding user intent, tool selection, and dynamic plan adjustment. Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes. In addressing these challenges, we investigate techniques within the context of fine-tuning and incontext learning paradigms, highlighting innovative approaches to ensure diversity, augment datasets, and improve generalization.Furthermore, we investigate a perspective on enabling LLMs to not only utilize but also autonomously create tools, which may redefine their role from mere tool users to tool creators. Finally,we reproduced Chameleon's results on ScienceQA and analyzed the code structure.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 30 of its 33 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

30 of 33 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error7 Mar
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  3. 2023GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution17 Jul
  4. 2023ToolQA: A Dataset for LLM Question Answering with External Tools23 Jun
  5. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  6. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  7. 2023Large Language Models as Tool Makers26 May
  8. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  9. 2023Making Language Models Better Tool Learners with Execution Feedback22 May
  10. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings19 May
  11. 2023Tool Learning with Foundation Models17 Apr
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Tools9 Feb

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  6. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 356
  7. 2021Internet-Augmented Dialogue Generationcited by 217
  8. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  9. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  10. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  11. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
  12. 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
6 more
  1. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  2. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 93
  3. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  4. 2023ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Modelscited by 43
  5. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Modelscited by 33
  6. 2023MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Promptingcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 38; 2026: 19.

61 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  3. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  5. 2025Agentic Large Language Models, a surveycited by 108
  6. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  7. 2025Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agentscited by 62
  8. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  9. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
  10. 2025MPMA: Preference Manipulation Attack Against Model Context Protocolcited by 34
  11. 2025API Agents vs. GUI Agents: Divergence and Convergencecited by 29
  12. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
38 more
  1. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  2. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  3. 2025From Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systemscited by 14
  4. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agentscited by 14
  5. 2025Accelerating Earth Science Discovery via Multi-Agent LLM Systemscited by 13
  6. 2026Dynamic Skill Lifecycle Management for Agentic Reinforcement Learningcited by 12
  7. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  8. 2025Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactionscited by 10
  9. 2025SEM: Reinforcement Learning for Search-Efficient Large Language Modelscited by 9
  10. 2024LLM-Virus: Evolutionary Jailbreak Attack on Large Language Modelscited by 7
  11. 2025A Large-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysiscited by 7
  12. 2025Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Callingcited by 7
  13. 2025PDE-Agent: A toolchain-augmented multi-agent framework for PDE solvingcited by 7
  14. 2025From Shots to Stories: LLM-Assisted Video Editing with Unified Language Representationscited by 6
  15. 2025Safe, Untrusted, "Proof-Carrying" AI Agents: toward the agentic lakehousecited by 6
  16. 2025Nested Browser-Use Learning for Agentic Information Seekingcited by 6
  17. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  18. 2025FunReason: Enhancing Large Language Models' Function Calling via Self-Refinement Multiscale Loss and Automated Data Refinementcited by 5
  19. 2025ToolRegistry: A Protocol-Agnostic Tool Management Library for Function-Calling LLMscited by 5
  20. 2025TaTToo: Tool-Grounded Thinking PRM for Test-Time Scaling in Tabular Reasoningcited by 5
  21. 2026Red Skills or Blue Skills? A Dive Into Skills Published on ClawHubcited by 5
  22. 2025Agentic AI for autonomous anomaly management in complex systemscited by 4
  23. 2026StreamingClaw Technical Reportcited by 4
  24. 2025InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agentscited by 3
  25. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltrationcited by 3
  26. 2026Physics-in-the-Loop: A Hybrid Agentic Architecture for Validated CAD Engineering Designcited by 3
  27. 2024Building Multi-Agent Copilot towards Autonomous Agricultural Data Management and Analysiscited by 2
  28. 2024LABIIUM: AI-Enhanced Zero-configuration Measurement Automation Systemcited by 2
  29. 2025ToPolyAgent: AI Agents for Coarse-Grained Topological Polymer Simulationscited by 2
  30. 2026Interfaze: The Future of AI is built on Task-Specific Small Modelscited by 2
  31. 2026CUBE: A Standard for Unifying Agent Benchmarkscited by 2
  32. 2026Tools as Continuous Flow for Evolving Agentic Reasoningcited by 2
  33. 2026Progressive Agent Skill Generation via Reinforcement Learningcited by 2
  34. 2025Addressing the sustainable AI trilemma: a case study on LLM agents and RAGcited by 1
  35. 2025Attentive Reasoning Queries: A Systematic Method for Optimizing Instruction-Following in Large Language Modelscited by 1
  36. 2025Efficient Serving of LLM Applications with Probabilistic Demand Modelingcited by 1
  37. 2026From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?cited by 1
  38. 2026TA-Mem: Tool-Augmented Autonomous Memory Retrieval for LLM in Long-Term Conversational QAcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Tool Learning with Large Language Models: A Survey
  2. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions
  3. 2024Language hooks: a modular framework for augmenting LLM reasoning that decouples tool usage from the model and its prompt
  4. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents
  5. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
  6. 2024Small LLMs Are Weak Tool Learners: A Multi-LLM Agent
  7. 2025TUMS: Enhancing Tool-use Abilities of LLMs with Multi-structure Handlers
  8. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios