mlsanityData to 7 Oct 2026

A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression

arXiv 2406.11430 ↗First posted 17 Jun 2024cs.CL cs.AIcited by 84

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The deployment of large language models (LLMs) is often hindered by the extensive memory requirements of the Key-Value (KV) cache, especially as context lengths increase. Existing approaches to reduce the KV cache size involve either fine-tuning the model to learn a compression strategy or leveraging attention scores to reduce the sequence length. We analyse the attention distributions in decoder-only Transformers-based models and observe that attention allocation patterns stay consistent across most layers. Surprisingly, we find a clear correlation between the L2 and the attention scores over cached KV pairs, where a low L2 of a key embedding usually leads to a high attention score during decoding. This finding indicates that the influence of a KV pair is potentially determined by the key embedding itself before being queried. Based on this observation, we compress the KV cache based on the L2 of key embeddings. Our experimental results show that this simple strategy can reduce the KV cache size by 50% on language modelling and needle-in-a-haystack tasks and 90% on passkey retrieval tasks without losing accuracy. Moreover, without relying on the attention scores, this approach remains compatible with FlashAttention, enabling broader applicability.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 38 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

30 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  10. 2023Vision Transformers Need Registerscited by 844
  11. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  12. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
14 more
  1. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  3. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  4. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  5. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  6. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  7. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  8. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  9. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  10. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  11. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
  12. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  13. 2024Analysing The Impact of Sequence Composition on Language Model Pre-Trainingcited by 22
  14. 2023Structured Packing in LLM Training Improves Long Context Utilizationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 34; 2026: 35.

84 papers.

Sort:

By area:

  1. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  4. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
  5. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  6. 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decodingcited by 65
  7. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 60
  8. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  9. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  10. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  11. 2025Cognitive Memory in Large Language Modelscited by 28
  12. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 28
38 more
  1. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 22
  2. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  3. 2025Attention Sinks in Diffusion Language Modelscited by 18
  4. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  5. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 15
  6. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Querycited by 15
  7. 2025The Pitfalls of KV Cache Compressioncited by 15
  8. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 13
  9. 2025AdaSplash: Adaptive Sparse Flash Attentioncited by 12
  10. 2026Learning to Evict from Key-Value Cachecited by 12
  11. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  12. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  13. 2025Let LLMs Speak Embedding Languages: Generative Text Embeddings via Iterative Contrastive Refinementcited by 9
  14. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Modelscited by 8
  15. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Importantcited by 7
  16. 2025QuickVideo: Real-Time Long Video Understanding with System Algorithm Co-Designcited by 7
  17. 2025Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMscited by 7
  18. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
  19. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 7
  20. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashingcited by 6
  21. 2024Cross-layer Attention Sharing for Pre-trained Large Language Modelscited by 5
  22. 2025KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embeddingcited by 5
  23. 2025GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Evictioncited by 5
  24. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoningcited by 5
  25. 2026MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inferencecited by 5
  26. 2024An Evolved Universal Transformer Memorycited by 4
  27. 2025Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compressioncited by 4
  28. 2025Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoningcited by 4
  29. 2025Scaling Test-Time Inference with Policy-Optimized, Dynamic Retrieval-Augmented Generation via KV Caching and Decodingcited by 4
  30. 2025OSCAR: Online Soft Compression And Rerankingcited by 4
  31. 2025Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Evictioncited by 4
  32. 2025TimeBill: Time-Budgeted Inference for Large Language Modelscited by 4
  33. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inferencecited by 4
  34. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objectivecited by 4
  35. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Frameworkcited by 3
  36. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  37. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  38. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SALS: Sparse Attention in Latent Space for KV cache Compression
  2. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
  3. 2025KV-CAR: KV Cache Compression using Autoencoders and KV Reuse in Large Language Models
  4. 2024Effectively Compress KV Heads for LLM
  5. 2025EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
  6. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
  7. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  8. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens