mlsanityData to 7 Oct 2026

What Are Tools Anyway? A Survey from the Language Model Perspective

arXiv 2403.15452 ↗First posted 18 Mar 2024cs.CL cs.AIcited by 53

Not on the map. Why?

Cited by 33 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models (LMs) are powerful yet mostly for text generation tasks. Tools have substantially enhanced their performance for tasks that require complex skills. However, many works adopt the term "tool" in different ways, raising the question: What is a tool anyway? Subsequently, where and how do tools help LMs? In this survey, we provide a unified definition of tools as external programs used by LMs, and perform a systematic review of LM tooling scenarios and approaches. Grounded on this review, we empirically study the efficiency of various tooling methods by measuring their required compute and performance gains on various benchmarks, and highlight some challenges and potential future research in the field.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 68 of its 88 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

68 of 88 references.

References · 68

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
38 more
  1. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  11. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  12. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  13. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  14. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  15. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  16. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  17. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  18. 2023Cognitive Architectures for Language Agentscited by 422
  19. 2023Augmented Language Models: a Surveycited by 398
  20. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  21. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  22. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  23. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  24. 2023Tool Learning with Foundation Modelscited by 302
  25. 2023Large Language Models as Tool Makerscited by 293
  26. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  27. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  28. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  29. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  30. 2022Binding Language Models in Symbolic Languagescited by 224
  31. 2021Internet-Augmented Dialogue Generationcited by 217
  32. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  33. 2022TALM: Tool Augmented Language Modelscited by 197
  34. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  35. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 159
  36. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  37. 2016DeepCoder: Learning to Write Programscited by 145
  38. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 138

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 28; 2026: 11.

53 papers.

Sort:

By area:

  1. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  2. 2024Agent Workflow Memorycited by 342
  3. 2024Tool Learning with Large Language Models: A Surveycited by 221
  4. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  5. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  6. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  7. 2024Beyond Browsing: API-Based Web Agentscited by 77
  8. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 67
  9. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 66
  10. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 47
  11. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  12. 2025Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Modelscited by 46
38 more
  1. 2024FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Modelscited by 39
  2. 2024FIRE: Fact-checking with Iterative Retrieval and Verificationcited by 39
  3. 2025Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learningcited by 22
  4. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  5. 2025RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solvingcited by 19
  6. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 17
  7. 2024A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Modelscited by 12
  8. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 11
  9. 2024TART: An Open-Source Tool-Augmented Framework for Explainable Table-based Reasoningcited by 10
  10. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  11. 2025Benchmarking Failures in Tool-Augmented Language Modelscited by 9
  12. 2025Orchestrating Agents and Data for Enterprise: A Blueprint Architecture for Compound AIcited by 9
  13. 2024Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?cited by 8
  14. 2024Adapting While Learning: Grounding LLMs for Scientific Problems with Intelligent Tool Usage Adaptationcited by 7
  15. 2025Self-Training Large Language Models for Tool-Use Without Demonstrationscited by 7
  16. 2025Self-Guided Function Calling in Large Language Models via Stepwise Experience Recallcited by 7
  17. 2025Procedural Knowledge Improves Agentic LLM Workflowscited by 7
  18. 2025PDE-Agent: A toolchain-augmented multi-agent framework for PDE solvingcited by 7
  19. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 4
  20. 2025Code-enabled language models can outperform reasoning models on diverse taskscited by 4
  21. 2025Verification-Guided Context Optimization for Tool Calling via Hierarchical LLMs-as-Editorscited by 3
  22. 2024A Survey on Complex Tasks for Goal-Directed Interactive Agentscited by 2
  23. 2025A Compute-Matched Re-Evaluation of TroVE on MATHcited by 2
  24. 2025The Social Cost of Intelligence: Emergence, Propagation, and Amplification of Stereotypical Bias in Multi-Agent Systemscited by 2
  25. 2025Reliable agent engineering should integrate machine-compatible organizational principlescited by 2
  26. 2026Cognitive Models and AI Algorithms Provide Templates for Designing Language Agentscited by 2
  27. 2026MistyPilot: An Agentic Fast-Slow Thinking LLM Framework for Misty Social Robotscited by 2
  28. 2026PhoenixRepair: Rethinking Repair Strategy Exploration in Software Agentscited by 2
  29. 2024Scaffolded Language Models with Language Supervision for Mixed-Autonomy: A Surveycited by 1
  30. 2025CyberMentor: AI Powered Learning Tool Platform to Address Diverse Student Needs in Cybersecurity Educationcited by 1
  31. 2025Multi-agent Application System in Office Collaboration Scenarioscited by 1
  32. 2025Assertion-Conditioned Compliance: A Provenance-Aware Vulnerability in Multi-Turn Tool-Calling Agentscited by 1
  33. 2026Exploring the Meta-level Reasoning of Large Language Models via a Tool-based Multi-hop Tabular Question Answering Taskcited by 1
  34. 2026SGA-MCTS: Decoupling Planning from Execution via Training-Free Atomic Experience Retrievalcited by 1
  35. 2026LLM Agents for Time-Series: A Surveycited by 1
  36. 2025SWI: Speaking with Intent in Large Language Modelscited by 0
  37. 2025From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Modelscited by 0
  38. 2025It's LIT! Reliability-Optimized LLMs with Inspectable Toolscited by 0

50 most cited of 53 All 53 (JSON)