# Tool-to-Agent Retrieval: Bridging Tools and Agents for Scalable LLM Multi-Agent Systems

arXiv: https://arxiv.org/abs/2511.01854 · HTML: https://mlsanity.com/paper/2511.01854/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Placed in

- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 30 of 30)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Graph Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2408.08921/) (2024)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph](https://mlsanity.com/paper/2307.07697/) (2023)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets](https://mlsanity.com/paper/2309.17428/) (2023)
- [AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls](https://mlsanity.com/paper/2402.04253/) (2024)
- [Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs](https://mlsanity.com/paper/2404.07103/) (2024)
- [HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction](https://mlsanity.com/paper/2408.04948/) (2024)
- [Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark](https://mlsanity.com/paper/2405.08355/) (2024)
- [ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph](https://mlsanity.com/paper/2403.00839/) (2024)
- [ControlLLM: Augment Language Models with Tools by Searching on Graphs](https://mlsanity.com/paper/2310.17796/) (2023)
- [ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario](https://mlsanity.com/paper/2501.10132/) (2025)
- [Re-Invoke: Tool Invocation Rewriting for Zero-Shot Tool Retrieval](https://mlsanity.com/paper/2408.01875/) (2024)
- [LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?](https://mlsanity.com/paper/2508.01780/) (2025)
- [ToolRerank: Adaptive and Hierarchy-Aware Reranking for Tool Retrieval](https://mlsanity.com/paper/2403.06551/) (2024)
- [ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agents](https://mlsanity.com/paper/2505.06416/) (2025)
- [MCP-Zero: Active Tool Discovery for Autonomous LLM Agents](https://mlsanity.com/paper/2506.01056/) (2025)
- [Leveraging Semantic and Lexical Matching to Improve the Recall of Document Retrieval Systems: A Hybrid Approach](https://mlsanity.com/paper/2010.01195/) (2020)
- [Toolshed: Scale Tool-Equipped Agents with Advanced RAG-Tool Fusion and Tool Knowledge Bases](https://mlsanity.com/paper/2410.14594/) (2024)
- [Planning and Editing What You Retrieve for Enhanced Tool Learning](https://mlsanity.com/paper/2404.00450/) (2024)
- [MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations](https://mlsanity.com/paper/2507.21428/) (2025)
- [REAPER: Reasoning based Retrieval Planning for Complex RAG Systems](https://mlsanity.com/paper/2407.18553/) (2024)
- [ProTIP: Progressive Tool Retrieval Improves Planning](https://mlsanity.com/paper/2312.10332/) (2023)

## Built on by (most cited first, 15 of 15)

- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [Self-Improvements in Modern Agentic Systems: A Survey](https://mlsanity.com/paper/2607.13104/) (2026)
- [Think like a Scientist: Physics-guided LLM Agent for Equation Discovery](https://mlsanity.com/paper/2602.12259/) (2026)
- [The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration](https://mlsanity.com/paper/2603.22862/) (2026)
- [Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents](https://mlsanity.com/paper/2605.06635/) (2026)
- [Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems](https://mlsanity.com/paper/2511.16654/) (2025)
- [Agent-as-a-Graph: Knowledge Graph-Based Tool and Agent Retrieval for LLM Multi-Agent Systems](https://mlsanity.com/paper/2511.18194/) (2025)
- [Automated Interpretability and Feature Discovery in Language Models with Agents](https://mlsanity.com/paper/2605.01555/) (2026)
- [Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?](https://mlsanity.com/paper/2605.07937/) (2026)
- [Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models](https://mlsanity.com/paper/2511.18177/) (2025)
- [GRAIL: A Deep-Granularity Hybrid Resonance Framework for Real-Time Agent Discovery via SLM-Enhanced Indexing](https://mlsanity.com/paper/2605.02489/) (2026)
- [Skill Description Deception Attack against Task Routing in Internet of Agents](https://mlsanity.com/paper/2605.09889/) (2026)
- [Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL](https://mlsanity.com/paper/2604.09470/) (2026)
- [A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization](https://mlsanity.com/paper/2606.30775/) (2026)
- [What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents](https://mlsanity.com/paper/2609.39989/) (2026)

## Abstract (arXiv)

Recent advances in LLM Multi-Agent Systems enable scalable orchestration of sub-agents, each coordinating hundreds or thousands of tools or Model Context Protocol (MCP) servers. However, existing retrieval methods typically match queries against coarse agent-level descriptions before routing, which obscures fine-grained tool functionality and often results in suboptimal agent selection. We introduce Tool-to-Agent Retrieval, a unified framework that embeds both tools and their parent agents in a shared vector space and connects them through metadata relationships. By explicitly representing tool capabilities and traversing metadata to the agent level, Tool-to-Agent Retrieval enables granular tool-level or agent-level retrieval, ensuring that agents and their underlying tools or MCP servers are equally represented without the context dilution that arises from chunking many tools together. Evaluating Tool-to-Agent Retrieval across eight embedding models, our approach achieves consistent improvements of 19.4% in Recall@5 and 17.7% in nDCG@5 over previous state-of-the-art agent retrievers on the LiveMCPBench benchmark.