mlsanityData to 14 Oct 2026

Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving

arXiv 2503.24000 ↗First posted 31 Mar 2025cs.LG cs.AIcited by 8

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Key-Value cache (textttKV textttcache) compression has emerged as a promising technique to optimize Large Language Model (LLM) serving. It primarily decreases the memory consumption of textttKV textttcache to reduce the computation cost. Despite the development of many compression algorithms, their applications in production environments are still not prevalent. In this paper, we revisit mainstream textttKV textttcache compression solutions from a practical perspective. Our contributions are three-fold. First, we comprehensively review existing algorithmic designs and benchmark studies for textttKV textttcache compression and identify missing pieces in their performance measurement, which could hinder their adoption in practice. Second, we empirically evaluate representative textttKV textttcache compression methods to uncover two key issues that affect the computational efficiency: (1) while compressing textttKV textttcache can reduce memory consumption, current implementations (e.g., FlashAttention, PagedAttention) do not optimize for production-level LLM serving, resulting in suboptimal throughput performance; (2) compressing textttKV textttcache may lead to longer outputs, resulting in increased end-to-end latency. We further investigate the accuracy performance of individual samples rather than the overall performance, revealing the intrinsic limitations in textttKV textttcache compression when handling specific LLM tasks. Third, we provide tools to shed light on future textttKV textttcache compression studies and facilitate their practical deployment in production. They are open-sourced in hrefhttps://github.com/LLMkvsys/rethink-kv-compressionhttps://github.com/LLMkvsys/rethink-kv-compression.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 52 of its 65 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

52 of 65 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction25 Sep
  3. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time7 Aug
  4. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  6. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  10. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  11. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
10 more
  1. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
  2. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  3. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  4. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  5. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  6. 2024On the Efficacy of Eviction Policy for Key-Value Constrained Generative Language Model Inference9 Feb
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  10. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023Mistral 7Bcited by 3,164
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  9. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
18 more
  1. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  2. 2024Transformers are Multi-State RNNscited by 120
  3. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  4. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  5. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  6. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  7. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  8. 2024Evaluating Quantized Large Language Modelscited by 77
  9. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  10. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  11. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  12. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  13. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Predictioncited by 66
  14. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  15. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 52
  16. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  17. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 44
  18. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  2. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  3. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  4. 2026Does Normalization Choice Matter for Causal Large Time-Series Models?cited by 0
  5. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agentscited by 0
  6. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Servingcited by 0
  7. 2026Divergence Timing and Cumulative Disagreement under KV-Cache Evictioncited by 0
  8. 2026EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  2. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  3. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  4. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  5. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  6. 2025AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
  7. 2025TreeKV: Smooth Key-Value Cache Compression with Tree Structures
  8. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache