# LLM With Tools: A Survey

arXiv: https://arxiv.org/abs/2409.18807 · HTML: https://mlsanity.com/paper/2409.18807/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 33 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models](https://mlsanity.com/paper/2304.09842/) (2023)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs](https://mlsanity.com/paper/2303.16434/) (2023)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [A Survey of Deep Learning for Mathematical Reasoning](https://mlsanity.com/paper/2212.10535/) (2022)
- [On the Tool Manipulation Capability of Open-source Large Language Models](https://mlsanity.com/paper/2305.16504/) (2023)
- [AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn](https://mlsanity.com/paper/2306.08640/) (2023)
- [WebCPM: Interactive Web Search for Chinese Long-form Question Answering](https://mlsanity.com/paper/2305.06849/) (2023)
- [CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models](https://mlsanity.com/paper/2305.14318/) (2023)
- [Making Language Models Better Tool Learners with Execution Feedback](https://mlsanity.com/paper/2305.13068/) (2023)
- [Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models](https://mlsanity.com/paper/2308.00675/) (2023)
- [ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models](https://mlsanity.com/paper/2305.14323/) (2023)
- [Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models](https://mlsanity.com/paper/2308.00304/) (2023)
- [MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Prompting](https://mlsanity.com/paper/2305.16896/) (2023)
- [LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error](https://mlsanity.com/paper/2403.04746/) (2024)
- [GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution](https://mlsanity.com/paper/2307.08775/) (2023)

## Built on by (most cited first, 30 of 61)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use](https://mlsanity.com/paper/2509.01055/) (2025)
- [Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents](https://mlsanity.com/paper/2507.22925/) (2025)
- [A Survey on Large Language Models for Mathematical Reasoning](https://mlsanity.com/paper/2506.08446/) (2025)
- [VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2509.19803/) (2025)
- [MPMA: Preference Manipulation Attack Against Model Context Protocol](https://mlsanity.com/paper/2505.11154/) (2025)
- [API Agents vs. GUI Agents: Divergence and Convergence](https://mlsanity.com/paper/2503.11069/) (2025)
- [LLM-Powered AI Agent Systems and Their Applications in Industry](https://mlsanity.com/paper/2505.16120/) (2025)
- [A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models](https://mlsanity.com/paper/2509.11575/) (2025)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [From Glue-Code to Protocols: A Critical Analysis of A2A and MCP Integration for Scalable Agent Systems](https://mlsanity.com/paper/2505.03864/) (2025)
- [ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agents](https://mlsanity.com/paper/2601.12294/) (2026)
- [Accelerating Earth Science Discovery via Multi-Agent LLM Systems](https://mlsanity.com/paper/2503.05854/) (2025)
- [Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning](https://mlsanity.com/paper/2605.10923/) (2026)
- [Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics](https://mlsanity.com/paper/2510.09901/) (2025)
- [Exposing LLM User Privacy via Traffic Fingerprint Analysis: A Study of Privacy Risks in LLM Agent Interactions](https://mlsanity.com/paper/2510.07176/) (2025)
- [SEM: Reinforcement Learning for Search-Efficient Large Language Models](https://mlsanity.com/paper/2505.07903/) (2025)
- [LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models](https://mlsanity.com/paper/2501.00055/) (2024)
- [A Large-Scale Evolvable Dataset for Model Context Protocol Ecosystem and Security Analysis](https://mlsanity.com/paper/2506.23474/) (2025)
- [Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling](https://mlsanity.com/paper/2508.05118/) (2025)
- [PDE-Agent: A toolchain-augmented multi-agent framework for PDE solving](https://mlsanity.com/paper/2512.16214/) (2025)
- [From Shots to Stories: LLM-Assisted Video Editing with Unified Language Representations](https://mlsanity.com/paper/2505.12237/) (2025)
- [Safe, Untrusted, "Proof-Carrying" AI Agents: toward the agentic lakehouse](https://mlsanity.com/paper/2510.09567/) (2025)
- [Nested Browser-Use Learning for Agentic Information Seeking](https://mlsanity.com/paper/2512.23647/) (2025)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [FunReason: Enhancing Large Language Models' Function Calling via Self-Refinement Multiscale Loss and Automated Data Refinement](https://mlsanity.com/paper/2505.20192/) (2025)

## Abstract (arXiv)

The integration of tools in augmenting large language models presents a novel approach toward enhancing the efficiency and accuracy of these models in handling specific, complex tasks. This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools, thereby pushing the boundaries of their capabilities beyond pre-existing knowledge bases. We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans and their execution, emphasizing the significance of understanding user intent, tool selection, and dynamic plan adjustment. Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes. In addressing these challenges, we investigate techniques within the context of fine-tuning and incontext learning paradigms, highlighting innovative approaches to ensure diversity, augment datasets, and improve generalization.Furthermore, we investigate a perspective on enabling LLMs to not only utilize but also autonomously create tools, which may redefine their role from mere tool users to tool creators. Finally,we reproduced Chameleon's results on ScienceQA and analyzed the code structure.