mlsanityData to 14 Oct 2026

Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models

arXiv 2503.01763 ↗First posted 3 Mar 2025cs.CL cs.AI cs.IRcited by 47

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Tool learning aims to augment large language models (LLMs) with diverse tools, enabling them to act as agents for solving practical tasks. Due to the limited context length of tool-using LLMs, adopting information retrieval (IR) models to select useful tools from large toolsets is a critical initial step. However, the performance of IR models in tool retrieval tasks remains underexplored and unclear. Most tool-use benchmarks simplify this step by manually pre-annotating a small set of relevant tools for each task, which is far from the real-world scenarios. In this paper, we propose ToolRet, a heterogeneous tool retrieval benchmark comprising 7.6k diverse retrieval tasks, and a corpus of 43k tools, collected from existing datasets. We benchmark six types of models on ToolRet. Surprisingly, even the models with strong performance in conventional IR benchmarks, exhibit poor performance on ToolRet. This low retrieval quality degrades the task pass rate of tool-use LLMs. As a further step, we contribute a large-scale training dataset with over 200k instances, which substantially optimizes the tool retrieval ability of IR models.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 42 of its 65 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

42 of 65 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactions10 Oct
  2. 2024ToolACE: Winning the Points of LLM Function Calling2 Sep
  3. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets26 Jun
  4. 2024Tool Learning with Large Language Models: A Survey28 May
  5. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning16 Jan
  6. 2023Reverse Chain: A Generic-Rule for LLMs to Master Multi-API Planning6 Oct
  7. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use4 Oct
  8. 2023Confucius: Iterative Tool Learning from Introspection Feedback by Easy-to-Difficult Curriculum27 Aug
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  10. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases8 Jun
  11. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction30 May
  12. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
1 more
  1. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs14 Apr

Other references · 29

  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2018Datasheets for Datasetscited by 1,386
  5. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  10. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  11. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  12. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
17 more
  1. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  2. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  3. 2024Generative Representational Instruction Tuningcited by 233
  4. 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
  5. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
  6. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  7. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 88
  8. 2024Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarioscited by 86
  9. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  10. 2024GTA: A Benchmark for General Tool Agentscited by 77
  11. 2024ToolGen: Unified Tool Retrieval and Calling via Generationcited by 74
  12. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 69
  13. 2024What Are Tools Anyway? A Survey from the Language Model Perspectivecited by 54
  14. 2024TableRAG: Million-Token Table Understanding with Language Modelscited by 41
  15. 2023Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Modelcited by 29
  16. 2024AppBench: Planning of Multiple APIs from Various APPs for Complex User Instructioncited by 24
  17. 2024MAIR: A Massive Benchmark for Evaluating Instructed Retrievalcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 39.

47 papers.

Sort:

By area:

  1. 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 71
  2. 2026SkillRouter: Skill Routing for LLM Agents at Scalecited by 61
  3. 2026Graph-of-Skills: Dependency-Aware Structural Retrieval for Massive Agent Skillscited by 55
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  5. 2026SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agentscited by 39
  6. 2026ReCreate: Reasoning and Creating Domain Agents Driven by Experiencecited by 24
  7. 2026A Comprehensive Survey on Agent Skills: Taxonomy, Techniques, and Applicationscited by 18
  8. 2026From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skillscited by 16
  9. 2026SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scalecited by 16
  10. 2025From REST to MCP: An Empirical Study of API Wrapping and Automated Server Generation for LLM Agentscited by 14
  11. 2026Group of Skills: Group-Structured Skill Retrieval for Agent Skill Librariescited by 11
  12. 2025Automated Composition of Agents: A Knapsack Approach for Agentic Component Selectioncited by 10
35 more
  1. 2026Diffusion-Pretrained Dense and Contextual Embeddingscited by 9
  2. 2026ToolChoiceConfusion: Causal Minimal Tool Filtering for Reliable LLM Agentscited by 9
  3. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  4. 2026LMEB: Long-horizon Memory Embedding Benchmarkcited by 7
  5. 2026How Many Tools Should an LLM Agent See? A Chance-Corrected Answercited by 7
  6. 2026Capability Minimization as a Safety Primitive: Risk-Aware Causal Gating for Least-Privilege LLM Agentscited by 6
  7. 2025ToolRegistry: A Protocol-Agnostic Tool Management Library for Function-Calling LLMscited by 5
  8. 2026Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performancecited by 5
  9. 2026Contract2Tool: Learning Preconditions and Effects for Reliable Tool-Augmented LLM Agentscited by 5
  10. 2026ToolMenuBench: Benchmarking Tool-Menu Filtering Strategies for Reliable and Efficient LLM Agentscited by 5
  11. 2025TheMCPCompany: Creating General-purpose Agents with Task-specific Toolscited by 4
  12. 2025LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluationcited by 4
  13. 2026Field-Aware Agent Skill Retrievalcited by 4
  14. 2026AgentSelect: Benchmark for Narrative Query-to-Agent Recommendationcited by 3
  15. 2026CaSKG: Counterfactual-Causal Skill Graphs for Scalable Agent Skill Retrievalcited by 3
  16. 2025ScaleCall -- Agentic Tool Calling at Scale for Fintech: Challenges, Methods, and Deployment Insightscited by 2
  17. 2026Multi-Field Tool Retrievalcited by 2
  18. 2026Skill2Query: Exploiting Skill Structure to Generate Pseudo-Queries for Agent Skill Retrievalcited by 2
  19. 2026Risk-Aware Reranking for Agentic Tool Retrievalcited by 2
  20. 2025In-N-Out: A Parameter-Level API Graph Dataset for Tool Agentscited by 1
  21. 2026AgentSearchBench: A Benchmark for AI Agent Search in the Wildcited by 1
  22. 2026ForecastAgentSearch: Towards a Multi-Expert Agent Search System for Geopolitical Event Forecastingcited by 1
  23. 2026MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Rerankingcited by 1
  24. 2026SE-GoS: Self-Evolving Graph-of-Skills for Skill Library at Scalecited by 1
  25. 2026The Router Within: Eliciting Native Skill Routing from a Frozen LLMcited by 1
  26. 2026What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Reviewcited by 1
  27. 2026Complete Cyclic Subtask Graphs for Tool-Using LLM Agents: Flexibility, Cost, and Bottlenecks in Long-Horizon Workflowscited by 0
  28. 2026FitText: Evolving Agent Tool Ecologies via Memetic Retrievalcited by 0
  29. 2026From Intent to Execution: Composing Agentic Workflows with Agent Recommendationcited by 0
  30. 2026When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agentscited by 0
  31. 2026String: An Agentic OS Where Every App Is a Markdown Filecited by 0
  32. 2026When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Usecited by 0
  33. 2026ParaAgent: Reinforcing Parallel Acting in Open-World Tool Environmentscited by 0
  34. 2026AnyAct: Universal Action for Self-Evolving Agentscited by 0
  35. 2026From Relevance to Validity: Zero-Shot Tool Selection via Functional and Contextual Validationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Tools are under-documented: Simple Document Expansion Boosts Tool Retrieval
  2. 2024Enhancing Tool Retrieval with Iterative Feedback from Large Language Models
  3. 2024Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models
  4. 2025MassTool: A Multi-Task Search-Based Tool Retrieval Framework for Large Language Models
  5. 2024Efficient and Scalable Estimation of Tool Representations in Vector Space
  6. 2024ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrieval
  7. 2024Towards Completeness-Oriented Tool Retrieval for Large Language Models
  8. 2026Semantic Tool Discovery for Large Language Models: A Vector-Based Approach to MCP Tool Selection