mlsanityData to 14 Oct 2026

Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management

arXiv 2609.06940 ↗First posted 7 Sep 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language model (LLM) inference increasingly spans models that differ in size, capability, price, and provider. This shift creates two costs for developers. One is the integration cost of choosing among and switching between many models. The other is the inference cost of rebuilding a KV cache when it is unavailable or incompatible with the selected model. We define and analyze the Unified AI Gateway as a system setting for an edge-deployed AI traffic hub. It coordinates model routing, KV cache management, and compute placement across end devices, edge resources, and cloud model services. At request time, the gateway jointly selects a target model, an execution site, and a KV cache action under task-quality, latency, cost, and resource constraints. In parallel, background cache-management actions optimize KV cache placement, replication, retrieval, and lifecycle decisions for subsequent requests. We synthesize existing evidence on KV cache reuse, compression, cross-model mapping, distributed storage, and transfer, and discuss the remaining challenges of integrating these capabilities into one system. Across eight typical workload profiles, our workload-level analytical simulation reports TTFT speedups of 1.25×--13.28× and input-cost benefits of 1.20×--6.16×.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 77 of its 91 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

77 of 91 references.

Closest ancestors · 31

Same direction, newest first

  1. 2026CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving20 Aug
  2. 2026Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse4 Aug
  3. 2026KV Cache Translation across Heterogeneous Large Language Models31 Jul
  4. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer30 Jul
  5. 2026SwiftCache: Efficient LLM Serving for Multi-turn Conversations with Heterogeneous KV Cache Sharing15 Jun
  6. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control7 Jun
  7. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention4 Jun
  8. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference18 May
  9. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse16 May
  10. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving13 May
  11. 2026SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving3 May
  12. 2026Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference19 Apr
19 more
  1. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter16 Apr
  2. 2026PrefixWall: Mitigating Prefix Caching Side Channels in Shared LLM Systems11 Mar
  3. 2026Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective26 Jan
  4. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale20 Dec
  5. 2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing16 Oct
  6. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  7. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Models3 Oct
  8. 2025SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matching29 Sep
  9. 2025Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference13 Aug
  10. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization26 May
  11. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse21 Feb
  12. 2025Online Scheduling for LLM Inference with KV Cache Constraints10 Feb
  13. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference5 Feb
  14. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  15. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  16. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  17. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  18. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  19. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  6. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  7. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  9. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  10. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
34 more
  1. 2017Non-Autoregressive Neural Machine Translationcited by 463
  2. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  3. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  4. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 70
  5. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  6. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  7. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  8. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systemscited by 40
  9. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  10. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 33
  11. 2024The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systemscited by 29
  12. 2026Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agentscited by 15
  13. 2026Agent Primitives: Reusable Latent Building Blocks for Multi-Agent Systemscited by 14
  14. 2025Efficient Training-Free Online Routing for High-Volume Multi-LLM Servingcited by 13
  15. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Schedulingcited by 10
  16. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  17. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agentscited by 10
  18. 2025Inference economics of language modelscited by 9
  19. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
  20. 2025Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRAcited by 8
  21. 2026ICaRus: Identical Cache Reuse for Efficient Multi Model Inferencecited by 8
  22. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reusecited by 8
  23. 2026See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agentscited by 8
  24. 2022Computing Power Network: A Surveycited by 7
  25. 2025Optimal-Agent-Selection: State-Aware Routing Framework for Efficient Multi-Agent Collaborationcited by 6
  26. 2025Hardware Acceleration for Neural Networks: A Comprehensive Surveycited by 6
  27. 2026Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patchingcited by 5
  28. 2026ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUscited by 3
  29. 2026Robust KV Cache Management for LLM Serving under Output Token Length Uncertaintycited by 3
  30. 2026Agent-Assisted Side-Channel Attacks on Non-Prefix KV Cache in RAGcited by 2
  31. 2026Towards Load-Aware Prefill Deflection for Disaggregated LLM Servingcited by 2
  32. 2026AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systemscited by 2
  33. 2026Towards Distributed Inference of LLMs on a P2P Networkcited by 1
  34. 2026The Price of Anarchy in Disaggregated Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
  2. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  3. 2026Joint Encoding of KV-Cache Blocks for Scalable LLM Serving
  4. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands
  5. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  6. 2024DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
  7. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  8. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructure