# CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing

arXiv: https://arxiv.org/abs/2508.16134 · HTML: https://mlsanity.com/paper/2508.16134/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference](https://mlsanity.com/paper/2410.21465/) (2024)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [Reducing Transformer Key-Value Cache Size with Cross-Layer Attention](https://mlsanity.com/paper/2405.12981/) (2024)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption](https://mlsanity.com/paper/2407.18003/) (2024)
- [KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing](https://mlsanity.com/paper/2410.18517/) (2024)
- [Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression](https://mlsanity.com/paper/2410.03765/) (2024)
- [TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention](https://mlsanity.com/paper/2410.05076/) (2024)
- [A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference](https://mlsanity.com/paper/2410.14442/) (2024)
- [ReAttention: Training-Free Infinite Context with Finite Attention Scope](https://mlsanity.com/paper/2407.15176/) (2024)
- [A Survey on Transformer Context Extension: Approaches and Evaluation](https://mlsanity.com/paper/2503.13299/) (2025)
- [Lossless KV Cache Compression to 2%](https://mlsanity.com/paper/2410.15252/) (2024)
- [Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache](https://mlsanity.com/paper/2506.11886/) (2025)

## Built on by (most cited first, 9 of 9)

- [OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization](https://mlsanity.com/paper/2605.17757/) (2026)
- [EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction](https://mlsanity.com/paper/2603.22910/) (2026)
- [Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing](https://mlsanity.com/paper/2604.22782/) (2026)
- [GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs](https://mlsanity.com/paper/2605.31105/) (2026)
- [Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering](https://mlsanity.com/paper/2601.11626/) (2026)
- [KV-Kaizen: Learning Context-Adaptive Cache Compression Choices](https://mlsanity.com/paper/2609.37988/) (2026)
- [S4R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching](https://mlsanity.com/paper/2608.00528/) (2026)
- [LatentIndex: Cross-Layer Sharing with Layer-Specific Selection for Sparse Attention](https://mlsanity.com/paper/2610.04635/) (2026)
- [VFold: Symmetry-Aware Cross-Layer Value Cache Compression](https://mlsanity.com/paper/2610.12338/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) confront significant memory challenges due to the escalating KV cache with increasing sequence length. As a crucial technique, existing cross-layer KV cache sharing methods either necessitate modified model architectures with subsequent pre-training or incur significant performance degradation at high compression rates. To mitigate these challenges, we propose CommonKV, a training-free method for cross-layer KV cache compression through adjacent parameters sharing. Inspired by the high similarity observed in cross-layer hidden states, we utilize Singular Value Decomposition (SVD) to achieve weight sharing across adjacent parameters, resulting in a more easily mergeable latent KV cache. Furthermore, we also introduce an adaptive budget allocation strategy. It dynamically assigns compression budgets based on cosine similarity, ensuring that dissimilar caches are not over-compressed. Experiments across multiple backbone models and benchmarks including LongBench and Ruler demonstrate that the proposed method consistently outperforms existing low-rank and cross-layer approaches at various compression ratios. Moreover, we find that the benefits of CommonKV are orthogonal to other quantization and eviction methods. By integrating these approaches, we can ultimately achieve a 98% compression ratio without significant performance loss.