Scalable LLM Agent Tool Access in the Cloud

arXiv 2607.15593 ↗First posted 17 Jul 2026cs.DC cs.AI cs.NIcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

LLM agents increasingly rely on tool calling to act on external systems, and the Model Context Protocol (MCP) has quickly become its de facto interface. Operating MCP at cloud scale, however, becomes difficult. On the tool provider side, legacy services are not directly callable through MCP; the rapid protocol development also creates ongoing compatibility cost. On the agent side, the number of accessible tool is limited by the LLM context window and inference overhead; mounting a large tool set increases token usage and inference latency and can reduce task success rate. Moreover, for stateful MCP backends with multiple replicas, preserving session affinity increases client-side complexity. We present a cloud-scale gateway system for MCP service. It breaks the direct-connect model on the data plane and offloads legacy service integration, consolidating incompatible MCP variants, access control, tool recommendation, and session-aware routing to the gateway. Hybrid retrieval sustains 98% Top-15 recall; it scales agent tool access to 3,000+ with high tool selection accuracy, and reduces tool selection time by 8.9× and token usage by 23.8×, with low per-call overhead, stable under scale-out. Finally, we share the lessons learned from deploying the gateway system in production.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 23 of its 55 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

23 of 55 references.

Closest ancestors · 2

Same direction, newest first

Other references · 21

2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
2024AgentScope: A Flexible yet Robust Multi-Agent Platformcited by 120
2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 96
2025A Survey of AI Agent Protocolscited by 77
2024NetLLM: Adapting Large Language Models for Networkingcited by 76
2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 57
2025LLM Agents Making Agent Toolscited by 45
2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32

9 more

2023Agent-OM: Leveraging LLM Agents for Ontology Matchingcited by 26
2025AgentScope 1.0: A Developer-Centric Framework for Building Agentic Applicationscited by 16
2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 15
2025A Measurement Study of Model Context Protocol Ecosystemcited by 13
2025Kairos: Low-latency Multi-Agent Serving with Shared LLMs and Excessive Loads in the Public Cloudcited by 11
2025Cortex: Achieving Low-Latency, Cost-Efficient Remote Data Access For LLM via Semantic-Aware Knowledge Cachingcited by 4
2023gDoc: Automatic Generation of Structured API Documentationcited by 2
2023Improving API Documentation Comprehensibility via Continuous Optimization and Multilingual SDKcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way