# What Are Tools Anyway? A Survey from the Language Model Perspective

arXiv: https://arxiv.org/abs/2403.15452 · HTML: https://mlsanity.com/paper/2403.15452/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 88 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 68)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)

## Built on by (most cited first, 30 of 54)

- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skills](https://mlsanity.com/paper/2504.07079/) (2025)
- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [Beyond Browsing: API-Based Web Agents](https://mlsanity.com/paper/2410.16464/) (2024)
- [SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents](https://mlsanity.com/paper/2508.02085/) (2025)
- [Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning](https://mlsanity.com/paper/2505.00024/) (2025)
- [ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows](https://mlsanity.com/paper/2505.19897/) (2025)
- [Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models](https://mlsanity.com/paper/2503.01763/) (2025)
- [From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions](https://mlsanity.com/paper/2410.08197/) (2024)
- [FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models](https://mlsanity.com/paper/2405.14767/) (2024)
- [FIRE: Fact-checking with Iterative Retrieval and Verification](https://mlsanity.com/paper/2411.00784/) (2024)
- [Graph-based Agent Memory: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2602.05665/) (2026)
- [Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning](https://mlsanity.com/paper/2506.04625/) (2025)
- [RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving](https://mlsanity.com/paper/2505.21577/) (2025)
- [How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations](https://mlsanity.com/paper/2510.22780/) (2025)
- [The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration](https://mlsanity.com/paper/2603.22862/) (2026)
- [A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Models](https://mlsanity.com/paper/2405.18208/) (2024)
- [TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoning](https://mlsanity.com/paper/2409.11724/) (2024)
- [Generalizability of Large Language Model-Based Agents: A Comprehensive Survey](https://mlsanity.com/paper/2509.16330/) (2025)
- [Benchmarking Failures in Tool-Augmented Language Models](https://mlsanity.com/paper/2503.14227/) (2025)
- [Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AI](https://mlsanity.com/paper/2504.08148/) (2025)
- [Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?](https://mlsanity.com/paper/2406.12307/) (2024)
- [Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptation](https://mlsanity.com/paper/2411.00412/) (2024)
- [Self-Training Large Language Models for Tool-Use Without Demonstrations](https://mlsanity.com/paper/2502.05867/) (2025)
- [Self-Guided Function Calling in Large Language Models via Stepwise Experience Recall](https://mlsanity.com/paper/2508.15214/) (2025)
- [Procedural Knowledge Improves Agentic LLM Workflows](https://mlsanity.com/paper/2511.07568/) (2025)
- [PDE-Agent: A toolchain-augmented multi-agent framework for PDE solving](https://mlsanity.com/paper/2512.16214/) (2025)

## Abstract (arXiv)

Language models (LMs) are powerful yet mostly for text generation tasks. Tools have substantially enhanced their performance for tasks that require complex skills. However, many works adopt the term "tool" in different ways, raising the question: What is a tool anyway? Subsequently, where and how do tools help LMs? In this survey, we provide a unified definition of tools as external programs used by LMs, and perform a systematic review of LM tooling scenarios and approaches. Grounded on this review, we empirically study the efficiency of various tooling methods by measuring their required compute and performance gains on various benchmarks, and highlight some challenges and potential future research in the field.