mlsanityData to 14 Oct 2026

Lossless KV Cache Compression to 2%

arXiv 2410.15252 ↗First posted 20 Oct 2024cs.CL cs.AIcited by 7

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models have revolutionized data processing in numerous domains, with their ability to handle extended context reasoning receiving notable recognition. To speed up inference, maintaining a key-value (KV) cache memory is essential. Nonetheless, the growing demands for KV cache memory create significant hurdles for efficient implementation. This work introduces a novel architecture, Cross-Layer Latent Attention (CLLA), aimed at compressing the KV cache to less than 2% of its original size while maintaining comparable performance levels. CLLA integrates multiple aspects of KV cache compression, including attention head/dimension reduction, layer sharing, and quantization techniques, into a cohesive framework. Our extensive experiments demonstrate that CLLA achieves lossless performance on most tasks while utilizing minimal KV cache, marking a significant advancement in practical KV cache compression.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 50 of its 65 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

50 of 65 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction25 Sep
  2. 2024Effectively Compress KV Heads for LLM11 Jun
  3. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  4. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  7. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb

Other references · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
31 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2023StarCoder: may the source be with you!cited by 799
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  13. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  14. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  15. 2021A White Paper on Neural Network Quantizationcited by 472
  16. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  17. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  18. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  19. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  20. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  21. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  22. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 142
  23. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  24. 2020OCNLI: Original Chinese Natural Language Inferencecited by 86
  25. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  26. 2020EasyQuant: Post-training Quantization via Scale Optimizationcited by 57
  27. 2024Learning to Compress Prompt in Natural Language Formatscited by 55
  28. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  29. 2024HMoE: Heterogeneous Mixture of Experts for Language Modelingcited by 29
  30. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 21
  31. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compressioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharingcited by 9
  3. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  4. 2025Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Spacecited by 5
  5. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  6. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 1
  7. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
  2. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  3. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  4. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  5. 2025SALS: Sparse Attention in Latent Space for KV cache Compression
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
  7. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
  8. 2026Kwai Summary Attention Technical Report