# A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference

arXiv: https://arxiv.org/abs/2410.14442 · HTML: https://mlsanity.com/paper/2410.14442/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 22 of 22)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [Reducing Transformer Key-Value Cache Size with Cross-Layer Attention](https://mlsanity.com/paper/2405.12981/) (2024)
- [The MiniPile Challenge for Data-Efficient Language Models](https://mlsanity.com/paper/2304.08442/) (2023)
- [Layer-Condensed KV Cache for Efficient Inference of Large Language Models](https://mlsanity.com/paper/2405.10637/) (2024)
- [MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding](https://mlsanity.com/paper/2406.09297/) (2024)
- [GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression](https://mlsanity.com/paper/2407.12077/) (2024)
- [Beyond KV Caching: Shared Attention for Efficient LLMs](https://mlsanity.com/paper/2407.12866/) (2024)
- [Skip-Layer Attention: Bridging Abstract and Detailed Dependencies in Transformers](https://mlsanity.com/paper/2406.11274/) (2024)

## Built on by (most cited first, 17 of 17)

- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Cache-to-Cache: Direct Semantic Communication Between Large Language Models](https://mlsanity.com/paper/2510.03215/) (2025)
- [Tensor Product Attention Is All You Need](https://mlsanity.com/paper/2501.06425/) (2025)
- [Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models](https://mlsanity.com/paper/2504.21038/) (2025)
- [CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing](https://mlsanity.com/paper/2508.16134/) (2025)
- [Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads](https://mlsanity.com/paper/2510.16807/) (2025)
- [Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing](https://mlsanity.com/paper/2604.22782/) (2026)
- [GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs](https://mlsanity.com/paper/2605.31105/) (2026)
- [YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference](https://mlsanity.com/paper/2604.13556/) (2026)
- [Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization](https://mlsanity.com/paper/2506.13541/) (2025)
- [TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference](https://mlsanity.com/paper/2508.15881/) (2025)
- [KV Cache Offloading for Context-Intensive Tasks](https://mlsanity.com/paper/2604.08426/) (2026)
- [KV-Kaizen: Learning Context-Adaptive Cache Compression Choices](https://mlsanity.com/paper/2609.37988/) (2026)
- [KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMs](https://mlsanity.com/paper/2608.15943/) (2026)
- [WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing](https://mlsanity.com/paper/2608.18486/) (2026)
- [Shared Global KV with Layer-Specific Local History](https://mlsanity.com/paper/2609.28006/) (2026)
- [Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers](https://mlsanity.com/paper/2610.13735/) (2026)

## Abstract (arXiv)

Recently, sharing key-value (KV) cache across layers has been found effective in efficient inference of large language models (LLMs). To systematically investigate different techniques of cross-layer KV sharing, we propose a unified framework that covers several recent methods and their novel variants. We conduct comprehensive experiments on all the configurations of the framework, evaluating their generation throughput and performance in language modeling and downstream tasks. We find that when reducing the size of the KV cache by 2×, most configurations can achieve higher throughput than standard transformers while maintaining competitive performance. When further reducing the size of the KV cache, however, pairing queries of all layers with KVs of upper layers performs better, at the expense of additional training cost and prefilling latency. We hope that this work will help users make more informed choices of cross-layer KV sharing approaches and facilitate future research on efficient LLM inference.