mlsanityData to 7 Oct 2026

MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding

arXiv 2406.09297 ↗First posted 13 Jun 2024cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Auto-regressive inference of transformers benefit greatly from Key-Value (KV) caching, but can lead to major memory bottlenecks as model size, batch size, and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV) sharing, a novel approach extending KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention (MQA) and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and inference metrics using uptrained Pythia-160M variants demonstrate that MLKV significantly reduces memory usage with minimal performance loss, reducing KV cache size down to a factor of 6x compared to MQA. These results highlight MLKV's potential for efficient deployment of transformer models at scale. We provide code at https://github.com/zaydzuhri/pythia-mlkv

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 18 of its 20 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

18 of 20 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
6 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  5. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  6. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 7; 2026: 6.

20 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025Cognitive Memory in Large Language Modelscited by 28
  3. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMscited by 27
  4. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  5. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
  6. 2024A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inferencecited by 16
  7. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 15
  8. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  9. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  10. 2024Lossless KV Cache Compression to 2%cited by 7
  11. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  12. 2024CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuningcited by 3
8 more
  1. 2024Art and Science of Quantizing Large-Scale Models: A Comprehensive Overviewcited by 2
  2. 2025Chain-of-Model Learning for Language Modelcited by 2
  3. 2024QET: Enhancing Quantized LLM Parameters and KV cache Compression through Element Substitution and Residual Clusteringcited by 0
  4. 2026Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Servingcited by 0
  5. 2026KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMscited by 0
  6. 2026WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixingcited by 0
  7. 2026The Extender: A Log-Structured Transformercited by 0
  8. 2026Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
  2. 2025End-to-End Transformer Acceleration Through Processing-in-Memory Architectures
  3. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models
  4. 2026Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
  5. 2026Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
  6. 2025Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers
  7. 2026The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference
  8. 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering