mlsanityData to 14 Oct 2026

KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider

arXiv 2506.02634 ↗First posted 3 Jun 2025cs.DC cs.AIcited by 41

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Serving large language models (LLMs) is important for cloud providers, and caching intermediate results (KV) after processing each request substantially improves serving throughput and latency. However, there is limited understanding of how LLM serving benefits from KV caching, where system design decisions like cache eviction policies are highly workload-dependent. In this paper, we present the first systematic characterization of the KV workload patterns from one of the leading LLM service providers. We draw observations that were not covered by previous studies focusing on synthetic workloads, including: KV reuses are skewed across requests, where reuses between single-turn requests are equally important as multi-turn requests; the reuse time and probability are diverse considering all requests, but for a specific request category, the pattern tends to be predictable; and the overall cache size required for an ideal cache hit ratio is moderate. Based on the characterization, we further propose a workload-aware cache eviction policy that improves the serving performance under real-world traces, especially with limited cache capacity.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 65 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

24 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Compute Or Load KV Cache? Why Not Both?4 Oct
  2. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  6. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  7. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  11. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  12. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
3 more
  1. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  2. 2023Fairness in Serving Large Language Modelscited by 78
  3. 2021Lightweight Robust Size Aware Cache Managementcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 30.

41 papers.

Sort:

By area:

  1. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  2. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 11
  3. 2026LLMServingSim 2.0: A Unified Simulator for Heterogeneous and Disaggregated LLM Serving Infrastructurecited by 9
  4. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  5. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  6. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  7. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 5
  8. 2026PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Servingcited by 4
  9. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  10. 2025Toward Robust and Efficient ML-Based GPU Caching for Modern Inferencecited by 3
  11. 2026FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Servingcited by 3
  12. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 3
29 more
  1. 2026FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inferencecited by 3
  2. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristicscited by 2
  3. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  4. 2025Revisiting Disaggregated Large Language Model Serving for Performance and Energy Implicationscited by 2
  5. 2026Simple is Better: Multiplication May Be All You Need for LLM Request Schedulingcited by 2
  6. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  7. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  8. 2026A Universal Context-Reuse Layer for Cross-Model KV Sharingcited by 2
  9. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 1
  10. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Servicecited by 1
  11. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1
  12. 2026Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Tracescited by 1
  13. 2026LaCache: Robust Semantic Caching for LLM Servingcited by 1
  14. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference servingcited by 1
  15. 2026HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Executioncited by 1
  16. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  17. 2026Lodestar: An Online-Learning LLM Inference Routercited by 0
  18. 2026Communication-Efficient Verifiable Attention for LLM Inferencecited by 0
  19. 2026Recency/Frequency Adaptive KV Caching for Large Language Model Servingcited by 0
  20. 2026Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Servingcited by 0
  21. 2026A Year in LLM Serving: Workload Evolution, Caching and Load-Balancingcited by 0
  22. 2026Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systemscited by 0
  23. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDscited by 0
  24. 2026Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundariescited by 0
  25. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systemscited by 0
  26. 2026DynBranch: Speculative Subgraph Reuse for Dynamic Agentic LLM Servingcited by 0
  27. 2026EfficientAgent: What Makes KV Cache Offloading Work for Concurrent Agents?cited by 0
  28. 2026PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decodingcited by 0
  29. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flashcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
  2. 2026Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective
  3. 2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Serving
  4. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
  5. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
  6. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  7. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
  8. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse