# RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2505.03275 · HTML: https://mlsanity.com/paper/2505.03275/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 6 of its 16 references that are in the map.
- First posted on arXiv 6 May 2025.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)
- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 6 of 6)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Enhancing Function-Calling Capabilities in LLMs: Strategies for Prompt Formats, Data Integration, and Multilingual Translation](https://mlsanity.com/paper/2412.01130/) (2024)

## Built on by (most cited first, 28 of 28)

- [Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolution](https://mlsanity.com/paper/2505.20286/) (2025)
- [Step-GUI Technical Report](https://mlsanity.com/paper/2512.15431/) (2025)
- [MCP-Zero: Active Tool Discovery for Autonomous LLM Agents](https://mlsanity.com/paper/2506.01056/) (2025)
- [More Skills, Worse Agents? Skill Shadowing Degrades Performance When Expanding Skill Libraries](https://mlsanity.com/paper/2605.24050/) (2026)
- [Code2MCP: Transforming Code Repositories into MCP Services](https://mlsanity.com/paper/2509.05941/) (2025)
- [Automated Composition of Agents: A Knapsack Approach for Agentic Component Selection](https://mlsanity.com/paper/2510.16499/) (2025)
- [Alita-G: Self-Evolving Generative Agent for Agent Generation](https://mlsanity.com/paper/2510.23601/) (2025)
- [ToolChoiceConfusion: Causal Minimal Tool Filtering for Reliable LLM Agents](https://mlsanity.com/paper/2606.06284/) (2026)
- [How Many Tools Should an LLM Agent See? A Chance-Corrected Answer](https://mlsanity.com/paper/2605.24660/) (2026)
- [Dynamic ReAct: Scalable Tool Selection for Large-Scale MCP Environments](https://mlsanity.com/paper/2509.20386/) (2025)
- [Agentic Services Computing](https://mlsanity.com/paper/2509.24380/) (2025)
- [Capability Minimization as a Safety Primitive: Risk-Aware Causal Gating for Least-Privilege LLM Agents](https://mlsanity.com/paper/2606.13884/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [TheMCPCompany: Creating General-purpose Agents with Task-specific Tools](https://mlsanity.com/paper/2510.19286/) (2025)
- [ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox](https://mlsanity.com/paper/2605.10787/) (2026)
- [Optimal Skill Selection for LLM Agents with Provable Bicriteria Guarantees](https://mlsanity.com/paper/2608.19993/) (2026)
- [Overwhelmed by Choice: Studying LLM Decision Making at Scale](https://mlsanity.com/paper/2609.32809/) (2026)
- [HuggingR4: A Progressive Reasoning Framework for Discovering Optimal Model Companions](https://mlsanity.com/paper/2511.18715/) (2025)
- [Z-Space: A Multi-Agent Tool Orchestration Framework for Enterprise-Grade LLM Automation](https://mlsanity.com/paper/2511.19483/) (2025)
- [MCP Server Architecture Patterns for LLM-Integrated Applications](https://mlsanity.com/paper/2606.30317/) (2026)
- [Mitigating Errors in LLM-Generated Web API Invocations via Retrieval-Augmented Generation and Constrained Decoding](https://mlsanity.com/paper/2607.05936/) (2026)
- [Aleena: Alignment Agent for Research Software Engineering Collaborations](https://mlsanity.com/paper/2607.08043/) (2026)
- [Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory](https://mlsanity.com/paper/2608.20397/) (2026)
- [Extracting Knowledge from Tools in LLM Agents](https://mlsanity.com/paper/2608.30288/) (2026)
- [Progressive Skill Discovery as Access Control for Tool-Using LLM Agents: Structural Governance through Role-Scoped Capability Delivery](https://mlsanity.com/paper/2609.28693/) (2026)
- [AnyAct: Universal Action for Self-Evolving Agents](https://mlsanity.com/paper/2609.37025/) (2026)
- [Which Metadata Matters for LLM Tool Selection in MCP? Evidence from Controlled Factor-Level Interventions](https://mlsanity.com/paper/2610.14334/) (2026)
- [Google Earth AI Geospatial Reasoning Agent: Capabilities and a Systematic Evaluation](https://mlsanity.com/paper/2610.15750/) (2026)

## Abstract (arXiv)

Large language models (LLMs) struggle to effectively utilize a growing number of external tools, such as those defined by the Model Context Protocol (MCP)citeIntroducingMCP, due to prompt bloat and selection complexity. We introduce RAG-MCP, a Retrieval-Augmented Generation framework that overcomes this challenge by offloading tool discovery. RAG-MCP uses semantic retrieval to identify the most relevant MCP(s) for a given query from an external index before engaging the LLM. Only the selected tool descriptions are passed to the model, drastically reducing prompt size and simplifying decision-making. Experiments, including an MCP stress test, demonstrate RAG-MCP significantly cuts prompt tokens (e.g., by over 50%) and more than triples tool selection accuracy (43.13% vs 13.62% baseline) on benchmark tasks. RAG-MCP enables scalable and accurate tool integration for LLMs.