mlsanityData to 14 Oct 2026

KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

arXiv 2605.13734 ↗First posted 13 May 2026cs.DC cs.AI cs.NIcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

LLMs are widely adopted in production, pushing inference systems to their limits. Disaggregated LLM serving (e.g., PD separation and KV state disaggregation) improves scalability and cost efficiency, but it also turns KV into an explicit payload crossing network and storage boundaries, making KV a dominant end-to-end bottleneck. Existing KV compression are typically static runtime configurations, despite production service context varies over time in workload mix, bandwidth, and SLO/quality budgets. As a result, a fixed choice can be suboptimal or even increase latency. We present KVServe, the first service-aware and adaptive KV communication compression framework for disaggregated LLM serving: KVServe (1) unifies KV compression into a modular strategy space with new components and cross-method recomposition; (2) introduces Bayesian Profiling Engine that efficiently searches this space and distills a 3D Pareto candidate set, reducing 50× offline search overhead; and (3) deploys a Service-Aware Online Controller that combines an analytical latency model with a lightweight bandit to select profiles under constraints and correct offline-to-online mismatch. Integrated into vLLM and evaluated across datasets, models, GPUs and networks, KVServe achieves up to 9.13× JCT speedup in PD-separated serving and up to 32.8× TTFT reduction in KV-disaggregated serving.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 53 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

32 of 53 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter16 Apr
  2. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruning12 Jan
  3. 2025KV Cache Transform Coding for Compact Storage in LLM Inference3 Nov
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  5. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  6. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache30 Aug
  7. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference5 Feb
  8. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  11. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  12. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 52
8 more
  1. 2025Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMscited by 42
  2. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 24
  3. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 22
  4. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  5. 2021BAGUA: Scaling up Distributed Learning with System Relaxationscited by 13
  6. 2025When to Reason: Semantic Router for vLLMcited by 12
  7. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  8. 2023gZCCL: Compression-Accelerated Collective Communication Framework for GPU Clusterscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Servingcited by 1
  2. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  3. 2026AFD-Ledger: Deployment Provisioning for Attention--FFN Disaggregationcited by 0
  4. 2026Read Less, Solve More: Token-Efficient Sparse Reading for AI Agentscited by 0
  5. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  6. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Servingcited by 0
  7. 2026LAM: Efficient Lossy Agent Memory Framework With A Retrieval-Score Error Boundcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving
  2. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  3. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  4. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  5. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service
  6. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
  7. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  8. 2026SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference