CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Tool use in LLM systems faces a structural trade-off. Progressive disclosure keeps the prompt small by showing only the tools relevant to the current task, while prompt caching rewards a request prefix that stays fixed across calls; every change to the visible tool list invalidates the cached prefix. This paper treats the trade-off as a problem of request architecture and proposes a dual-path routing design that assigns tool selection and tool delivery to separate channels. The main model always sees a small, fixed set of core tools, so the head of its request is unchanged across calls; all other tools are reached through an independent routing channel, in which a router sub-model searches the full tool list, selects one tool, executes it, and returns the result. Tool registration is automated from source code and supports runtime updates, so the tool set can grow without modifying the main model's request prefix. The design generalizes progressive disclosure: capabilities are disclosed through the routing channel, and the main model's prefix stays stable. A prototype implementation was exercised on 55 functional queries and a 30-turn dialogue; token-level cache hit rates reached 90.99% and 95.2%, cutting input cost to about 12.0% and 8.0% of a no-cache baseline under DeepSeek's pricing, where cache-hit input tokens cost roughly 1/30 of cache-miss tokens.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 25 references that are in the map. First posted on arXiv 24 Aug 2026.
Builds on
17 of 25 references.
References · 17
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2023Large Language Models as Tool Makerscited by 293
- 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
5 more
- 2024Tool Learning with Large Language Models: A Surveycited by 221
- 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
- 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
- 2023An LLM Compiler for Parallel Function Callingcited by 170
- 2024AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Callscited by 121
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026ToolCaching: Towards Efficient Caching for LLM Tool-calling
- 2024Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution
- 2026TokenRouter: Efficient Serving System for Token-Level LLM Routing
- 2026Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory
- 2026TVCACHE: A Stateful Tool-Value Cache for Post-Training LLM Agents
- 2026Latency-Quality Routing for Functionally Equivalent Tools in LLM Agents
- 2024LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching
- 2026CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving