mlsanityData to 7 Oct 2026

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

arXiv 2405.12981 ↗First posted 21 May 2024cs.LG cs.CLcited by 122

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Key-value (KV) caching plays an essential role in accelerating decoding for transformer-based autoregressive large language models (LLMs). However, the amount of memory required to store the KV cache can become prohibitive at long sequence lengths and large batch sizes. Since the invention of the transformer, two of the most effective interventions discovered for reducing the size of the KV cache have been Multi-Query Attention (MQA) and its generalization, Grouped-Query Attention (GQA). MQA and GQA both modify the design of the attention block so that multiple query heads can share a single key/value head, reducing the number of distinct key/value heads by a large factor while only minimally degrading accuracy. In this paper, we show that it is possible to take Multi-Query Attention a step further by also sharing key and value heads between adjacent layers, yielding a new attention design we call Cross-Layer Attention (CLA). With CLA, we find that it is possible to reduce the size of the KV cache by another 2x while maintaining nearly the same accuracy as unmodified MQA. In experiments training 1B- and 3B-parameter models from scratch, we demonstrate that CLA provides a Pareto improvement over the memory/accuracy tradeoffs which are possible with traditional MQA, enabling inference with longer sequence lengths and larger batch sizes than would otherwise be possible

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 39 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

24 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
9 more
  1. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2023A Survey on Model Compression for Large Language Modelscited by 263
  4. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  5. 2022Memorizing Transformerscited by 219
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  7. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  8. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  9. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 43; 2026: 44.

122 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  6. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  7. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  8. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  9. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  10. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  11. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  12. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
38 more
  1. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  2. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  3. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  4. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  5. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 38
  6. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  7. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  8. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  9. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  10. 2025MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connectionscited by 27
  11. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  12. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  13. 2024Value Residual Learningcited by 25
  14. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 20
  15. 2025GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignmentcited by 20
  16. 2024A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inferencecited by 16
  17. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
  18. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 15
  19. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 15
  20. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 14
  21. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
  22. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  23. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  24. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  25. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similaritycited by 11
  26. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  27. 2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 11
  28. 2024Beyond KV Caching: Shared Attention for Efficient LLMscited by 10
  29. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  30. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  31. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationcited by 10
  32. 2026Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reusecited by 10
  33. 2024KV Prediction for Improved Time to First Tokencited by 9
  34. 2025Ladder-residual: parallelism-aware architecture for accelerating large model inference with communication overlappingcited by 9
  35. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 9
  36. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharingcited by 8
  37. 2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 8
  38. 2026HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharingcited by 8

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
  2. 2024Effectively Compress KV Heads for LLM
  3. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression
  4. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
  5. 2026ART: Attention Run-time Termination for Efficient Large Language Model Decoding
  6. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
  7. 2026Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection
  8. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation