mlsanityData to 7 Oct 2026

xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction

arXiv 2503.18893 ↗First posted 24 Mar 2025cs.CL cs.LGcited by 13

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context Large Language Models (LLMs) enable powerful applications but incur high memory costs due to the key-value states (KV-Cache). Recent studies attempt to share KV-Cache across layers, but these approaches either require expensive pretraining or rely on per-token cross-layer cosine similarity that is often limited in practice. We show, via Centered Kernel Alignment (CKA), that the dominant singular vectors of KV-Cache are well aligned across layers. Motivated by this observation, we propose xKV, a post-training compression method that jointly factorizes grouped-layer KV-Cache into a shared low-rank subspace, substantially reducing KV-Cache memory. Across widely used LLMs, xKV achieves up to 8x KV-Cache compression while preserving accuracy on long-context tasks and in multi-turn settings. To further improve efficiency, we introduce Selective Reconstruction (SR) at decode time. Combined with SR, xKV achieves up to 4.23x end-to-end speedup over the full attention baseline, and surpasses notable baselines with 30% higher throughput under a similar accuracy level. Overall, xKV provides a plug-and-play approach to reduce both memory and latency for long-context LLM inference. Our code is publicly available at: https://github.com/abdelfattah-lab/xKV.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 48 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

33 of 48 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025TokenButler: Token Importance is Predictable10 Mar
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec
  3. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  5. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  6. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  10. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  11. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
3 more
  1. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  12. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
6 more
  1. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  2. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  3. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  4. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  5. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  6. 2025RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remappingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  2. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  3. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
  4. 2026Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoningcited by 2
  5. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  6. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  7. 2026A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocationcited by 0
  8. 2026S4R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Cachingcited by 0
  9. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compressioncited by 0
  10. 2026Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparisoncited by 0
  11. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 0
  12. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0
1 more
  1. 2026AttSVD:Prompt-Adaptive Low-Rank KV Cache Compression via Attention-Guided SVDcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing
  2. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
  3. 2024CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios
  4. 2026DynaCalKV: Key-Value Cache Compression via Head Grouping and Adaptive Rank Allocation
  5. 2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMs
  6. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  7. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  8. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs