mlsanityData to 14 Oct 2026

Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniques

arXiv 2503.11816 ↗First posted 14 Mar 2025cs.CLcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) have demonstrated exceptional capabilities in generating text, images, and video content. However, as context length grows, the computational cost of attention increases quadratically with the number of tokens, presenting significant efficiency challenges. This paper presents an analysis of various Key-Value (KV) cache compression strategies, offering a comprehensive taxonomy that categorizes these methods by their underlying principles and implementation techniques. Furthermore, we evaluate their impact on performance and inference latency, providing critical insights into their effectiveness. Our findings highlight the trade-offs involved in KV cache compression and its influence on handling long-context scenarios, paving the way for more efficient LLM implementations.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 49 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

46 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct

Other references · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
33 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2015Teaching Machines to Read and Comprehendcited by 1,049
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  13. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  14. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  15. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  16. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  17. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  18. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  19. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  20. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  21. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  22. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  23. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  24. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  25. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  26. 2019On the Use of ArXiv as a Datasetcited by 103
  27. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 90
  28. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  29. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  30. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  31. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inferencecited by 53
  32. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  33. 2023TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reductioncited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025AI Compute Architecture and Evolution Trendscited by 1
  2. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 1
  3. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placementcited by 0
  4. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  5. 2026EdgeLLM-Nano: Auditing Polar-Guided Key-Cache Compression and Consistency-Anchored Pruning on a 33.6 MB On-Device Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving
  2. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  3. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  4. 2025KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache
  5. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
  8. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption