mlsanityData to 7 Oct 2026

On the Tool Manipulation Capability of Open-source Large Language Models

arXiv 2305.16504 ↗First posted 25 May 2023cs.CL cs.AI cs.LGcited by 122

Not on the map. Why?

Cited by 71 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent studies on software tool manipulation with large language models (LLMs) mostly rely on closed model APIs. The industrial adoption of these models is substantially constrained due to the security and robustness risks in exposing information to closed LLM API services. In this paper, we ask can we enhance open-source LLMs to be competitive to leading closed LLM APIs in tool manipulation, with practical amount of human supervision. By analyzing common tool manipulation failures, we first demonstrate that open-source LLMs may require training with usage examples, in-context demonstration and generation style regulation to resolve failures. These insights motivate us to revisit classical methods in LLM literature, and demonstrate that we can adapt them as model alignment with programmatic data generation, system prompts and in-context demonstration retrievers to enhance open-source LLMs for tool manipulation. To evaluate these techniques, we create the ToolBench, a tool manipulation benchmark consisting of diverse software tools for real-world tasks. We demonstrate that our techniques can boost leading open-source LLMs by up to 90% success rate, showing capabilities competitive to OpenAI GPT-4 in 4 out of 8 ToolBench tasks. We show that such enhancement typically requires about one developer day to curate data for each tool, rendering a recipe with practical amount of human supervision.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Builds on 45 of its 68 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

45 of 68 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
33 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  13. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  14. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  15. 2023StarCoder: may the source be with you!cited by 790
  16. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  17. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  18. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  19. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  20. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  21. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  22. 2023Augmented Language Models: a Surveycited by 398
  23. 2023Language Models can Solve Computer Taskscited by 371
  24. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  25. 2023Tool Learning with Foundation Modelscited by 302
  26. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  27. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  28. 2022TALM: Tool Augmented Language Modelscited by 197
  29. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  30. 2023SantaCoder: don't reach for the stars!cited by 169
  31. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  32. 2022A Survey on Retrieval-Augmented Text Generationcited by 157
  33. 2020MLIR: A Compiler Infrastructure for the End of Moore's Lawcited by 64

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 40; 2025: 26; 2026: 32.

122 papers.

Sort:

By area:

  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  3. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  4. 2023A Comprehensive Overview of Large Language Modelscited by 639
  5. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  6. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  7. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  8. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  9. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  10. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  11. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  12. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
38 more
  1. 2024Tool Learning with Large Language Models: A Surveycited by 221
  2. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  3. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  4. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  5. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 122
  8. 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
  9. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  10. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  11. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 66
  12. 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
  13. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculumcited by 64
  14. 2024LLM With Tools: A Surveycited by 60
  15. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 60
  16. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
  17. 2024Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?cited by 55
  18. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  19. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 53
  20. 2026Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skillscited by 53
  21. 2024Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Taskscited by 52
  22. 2023Do Large Language Models Know about Facts?cited by 49
  23. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  24. 2025MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Modelscited by 48
  25. 2023LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?cited by 47
  26. 2024WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Settingcited by 47
  27. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  28. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  29. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  30. 2024NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Callscited by 45
  31. 2023ToolTalk: Evaluating Tool-Usage in a Conversational Settingcited by 42
  32. 2024MIRAI: Evaluating LLM Agents for Event Forecastingcited by 40
  33. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systemscited by 39
  34. 2024ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agentscited by 36
  35. 2024Assessing and Understanding Creativity in Large Language Modelscited by 35
  36. 2024Efficient Tool Use with Chain-of-Abstraction Reasoningcited by 33
  37. 2024CACTUS: Chemistry Agent Connecting Tool-Usage to Sciencecited by 32
  38. 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents
  2. 2026Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives
  3. 2024Achieving Tool Calling Functionality in LLMs Using Only Prompt Engineering Without Fine-Tuning
  4. 2024ToolScan: A Benchmark for Characterizing Errors in Tool-Use LLMs
  5. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
  6. 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agents
  7. 2024From Summary to Action: Enhancing Large Language Models for Complex Tasks with Open World APIs
  8. 2023Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Model