mlsanityData to 14 Oct 2026

A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference

arXiv 2410.14442 ↗First posted 18 Oct 2024cs.CLcited by 17

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Recently, sharing key-value (KV) cache across layers has been found effective in efficient inference of large language models (LLMs). To systematically investigate different techniques of cross-layer KV sharing, we propose a unified framework that covers several recent methods and their novel variants. We conduct comprehensive experiments on all the configurations of the framework, evaluating their generation throughput and performance in language modeling and downstream tasks. We find that when reducing the size of the KV cache by 2×, most configurations can achieve higher throughput than standard transformers while maintaining competitive performance. When further reducing the size of the KV cache, however, pairing queries of all layers with KVs of upper layers performs better, at the expense of additional training cost and prefilling latency. We hope that this work will help users make more informed choices of cross-layer KV sharing approaches and facilitate future research on efficient LLM inference.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 28 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

22 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression16 Jul
  2. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  3. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 18

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
6 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  2. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  3. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  4. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 21
  5. 2024Beyond KV Caching: Shared Attention for Efficient LLMscited by 10
  6. 2024Skip-Layer Attention: Bridging Abstract and Detailed Dependencies in Transformerscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 9.

17 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 61
  3. 2025Tensor Product Attention Is All You Needcited by 31
  4. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Modelscited by 10
  5. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharingcited by 9
  6. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  7. 2026Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharingcited by 4
  8. 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMscited by 4
  9. 2026YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inferencecited by 2
  10. 2025Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimizationcited by 1
  11. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  12. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
5 more
  1. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 1
  2. 2026KV-Pipe: On the Relation Between KV Sharing and Pipeline Parallel Efficiency in LLMscited by 0
  3. 2026WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixingcited by 0
  4. 2026Shared Global KV with Layer-Specific Local Historycited by 0
  5. 2026Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  3. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  5. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  6. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
  7. 2026River-LLM: Large Language Model Seamless Exit Based on KV Share
  8. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing