# MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding

arXiv: https://arxiv.org/abs/2406.09297 · HTML: https://mlsanity.com/paper/2406.09297/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 20 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [Reducing Transformer Key-Value Cache Size with Cross-Layer Attention](https://mlsanity.com/paper/2405.12981/) (2024)

## Built on by (most cited first, 21 of 21)

- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse](https://mlsanity.com/paper/2603.12201/) (2026)
- [Cognitive Memory in Large Language Models](https://mlsanity.com/paper/2504.02441/) (2025)
- [Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs](https://mlsanity.com/paper/2502.14837/) (2025)
- [TransMLA: Multi-Head Latent Attention Is All You Need](https://mlsanity.com/paper/2502.07864/) (2025)
- [Accurate KV Cache Quantization with Outlier Tokens Tracing](https://mlsanity.com/paper/2505.10938/) (2025)
- [A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference](https://mlsanity.com/paper/2410.14442/) (2024)
- [More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression](https://mlsanity.com/paper/2412.12706/) (2024)
- [Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures](https://mlsanity.com/paper/2508.10824/) (2025)
- [Lossless KV Cache Compression to 2%](https://mlsanity.com/paper/2410.15252/) (2024)
- [Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads](https://mlsanity.com/paper/2510.16807/) (2025)
- [CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuning](https://mlsanity.com/paper/2411.17426/) (2024)
- [Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview](https://mlsanity.com/paper/2409.11650/) (2024)
- [Chain-of-Model Learning for Language Model](https://mlsanity.com/paper/2505.11820/) (2025)
- [QET: Enhancing Quantized LLM Parameters and KV cache Compression through Element Substitution and Residual Clustering](https://mlsanity.com/paper/2407.03637/) (2024)
- [Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving](https://mlsanity.com/paper/2607.05399/) (2026)
- [KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMs](https://mlsanity.com/paper/2608.15943/) (2026)
- [WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing](https://mlsanity.com/paper/2608.18486/) (2026)
- [The Extender: A Log-Structured Transformer](https://mlsanity.com/paper/2609.32759/) (2026)
- [LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention](https://mlsanity.com/paper/2610.04635/) (2026)
- [Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression](https://mlsanity.com/paper/2610.10552/) (2026)

## Abstract (arXiv)

Auto-regressive inference of transformers benefit greatly from Key-Value (KV) caching, but can lead to major memory bottlenecks as model size, batch size, and sequence length grow at scale. We introduce Multi-Layer Key-Value (MLKV) sharing, a novel approach extending KV sharing across transformer layers to reduce memory usage beyond what was possible with Multi-Query Attention (MQA) and Grouped-Query Attention (GQA). Evaluations on various NLP benchmarks and inference metrics using uptrained Pythia-160M variants demonstrate that MLKV significantly reduces memory usage with minimal performance loss, reducing KV cache size down to a factor of 6x compared to MQA. These results highlight MLKV's potential for efficient deployment of transformer models at scale. We provide code at https://github.com/zaydzuhri/pythia-mlkv