mlsanityData to 7 Oct 2026

Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption

arXiv 2407.18003 ↗First posted 25 Jul 2024cs.CLcited by 96

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs), epitomized by ChatGPT's release in late 2022, have revolutionized various industries with their advanced language comprehension. However, their efficiency is challenged by the Transformer architecture's struggle with handling long texts. KV Cache has emerged as a pivotal solution to this issue, converting the time complexity of token generation from quadratic to linear, albeit with increased GPU memory overhead proportional to conversation length. With the development of the LLM community and academia, various KV Cache compression methods have been proposed. In this review, we dissect the various properties of KV Cache and elaborate on various methods currently used to optimize the KV Cache space usage of LLMs. These methods span the pre-training phase, deployment phase, and inference phase, and we summarize the commonalities and differences among these methods. Additionally, we list some metrics for evaluating the long-text capabilities of large language models, from both efficiency and capability perspectives. Our review thus sheds light on the evolving landscape of LLM optimization, offering insights into future advancements in this dynamic field. Links to the papers mentioned in this review can be found in our Github Repo https://github.com/zcli-charlie/Awesome-KV-Cache.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 62 of its 72 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

62 of 72 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  2. 2024GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression16 Jul
  3. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression17 Jun
  6. 2024Effectively Compress KV Heads for LLM11 Jun
  7. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  8. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  10. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  11. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  12. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
7 more
  1. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition23 Feb
  2. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  3. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  4. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  10. 2024Yi: Open Foundation Models by 01.AIcited by 706
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
31 more
  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  4. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  5. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  6. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  7. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  8. 2023Adapting Language Models to Compress Contextscited by 350
  9. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  10. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  12. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  13. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  14. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  15. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  16. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  17. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  18. 2024Transformers are Multi-State RNNscited by 120
  19. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  20. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  21. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  22. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  23. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  24. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  25. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  26. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  27. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  28. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  29. 2024XL2Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependenciescited by 14
  30. 2024Anchor-based Large Language Modelscited by 12
  31. 2020Revisiting Linformer with a modified self-attention with linear complexitycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 59; 2026: 25.

96 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  4. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  5. 2024LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Acceleratorscited by 31
  6. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  7. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  8. 2025Taming Teacher Forcing for Masked Autoregressive Video Generationcited by 29
  9. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  10. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  11. 2024SCOPE: Optimizing Key-Value Cache Compression in Long-context Generationcited by 20
  12. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
38 more
  1. 2026Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Taskscited by 18
  2. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
  3. 2024Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devicescited by 15
  4. 2025TTS-VAR: A Test-Time Scaling Framework for Visual Auto-Regressive Generationcited by 15
  5. 2025The Pitfalls of KV Cache Compressioncited by 15
  6. 2025AdaSplash: Adaptive Sparse Flash Attentioncited by 12
  7. 2025MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inferencecited by 12
  8. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  9. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  10. 2025Inference-time sparse attention with asymmetric indexingcited by 11
  11. 2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
  12. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11
  13. 2025EvolKV: Evolutionary KV Cache Compression for LLM Inferencecited by 10
  14. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 9
  15. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Modelscited by 9
  16. 2025SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layerscited by 9
  17. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelitycited by 9
  18. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 8
  19. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inferencecited by 8
  20. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharingcited by 8
  21. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  22. 2025Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitivescited by 7
  23. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  24. 2025DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compressioncited by 7
  25. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
  26. 2025HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Modelscited by 6
  27. 2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
  28. 2025OjaKV: Context-Aware Online Low-Rank KV Cache Compressioncited by 6
  29. 2025SemShareKV: Efficient KVCache Sharing for Semantically Similar Prompts via Token-Level LSH Matchingcited by 6
  30. 2025Compression Barriers for Autoregressive Transformerscited by 5
  31. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMscited by 5
  32. 2024An Evolved Universal Transformer Memorycited by 4
  33. 2025Cost-Optimal Grouped-Query Attention for Long-Context Modelingcited by 4
  34. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compressioncited by 4
  35. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4
  36. 2025EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agentcited by 4
  37. 2025CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Modelscited by 4
  38. 2025XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compressioncited by 4

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  3. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  4. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques
  5. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  6. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
  7. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
  8. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference