mlsanityData to 7 Oct 2026

Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

arXiv 2607.08057 ↗First posted 9 Jul 2026cs.LG cs.AI cs.CLcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache, which stores KV tensors during autoregressive decoding, is crucial for enabling low-latency, high-throughput LLM inference serving. In this survey, we focus on system-aware KV infrastructure for serving LLMs (abbreviated as sKis). We revisit recent work from a system behavior perspective, organizing existing efforts into three dimensions: execution and scheduling (temporal), placement and migration (spatial), and representation and retention (structural). Furthermore, we analyze cross-behavior co-design affinity and behavior-objective links, highlighting future opportunities. Our work systematizes a rapidly evolving area, providing a foundation for understanding and innovating KV cache designs in modern LLM serving infrastructure.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 127 of its 185 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

127 of 185 references.

Closest ancestors · 55

Same direction, newest first

  1. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inference8 Feb
  2. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale20 Dec
  3. 2025CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving11 Dec
  4. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management25 Nov
  5. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  6. 2025EvolKV: Evolutionary KV Cache Compression for LLM Inference10 Sep
  7. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens5 Sep
  8. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning8 Aug
  9. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review8 Aug
  10. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibration30 May
  11. 2025Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching8 Apr
  12. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences16 Mar
38 more
  1. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression19 Feb
  2. 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving14 Jan
  3. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  5. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
  6. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  7. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
  8. 2024RefreshKV: Updating Small KV Cache During Long-form Generation8 Nov
  9. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection5 Nov
  10. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference30 Oct
  11. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  12. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing24 Oct
  13. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  14. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management1 Oct
  15. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference8 Sep
  16. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  17. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  18. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  19. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  20. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  21. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving22 Jul
  22. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  23. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  24. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  25. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters18 Jun
  26. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  27. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  28. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  29. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  30. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  31. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  32. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  33. 2024CHAI: Clustered Head Attention for Efficient LLM Inference12 Mar
  34. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  35. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving4 Mar
  36. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  37. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  38. 2023Stateful Large Language Model Serving with Pensieve9 Dec

50 newest of 55 All 55 (JSON)

Other references · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
38 more
  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  4. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  6. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  7. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  8. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  9. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  10. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  11. 2019MLPerf Inference Benchmarkcited by 195
  12. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  13. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  14. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  15. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  16. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  17. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  18. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 110
  19. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  20. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  21. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  22. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  23. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  24. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  25. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  26. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  27. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  28. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  29. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  30. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  31. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  32. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  33. 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
  34. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  35. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
  36. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  37. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  38. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 31

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metricscited by 0
  2. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  3. 2025Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity
  4. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  5. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  6. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  7. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  8. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models