# SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention

arXiv: https://arxiv.org/abs/2502.15304 · HTML: https://mlsanity.com/paper/2502.15304/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache quantization](https://mlsanity.com/inference/kv-cache-quantization/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Efficiently Scaling Transformer Inference](https://mlsanity.com/paper/2211.05102/) (2022)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2403.07378/) (2024)
- [QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving](https://mlsanity.com/paper/2405.04532/) (2024)
- [SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models](https://mlsanity.com/paper/2411.05007/) (2024)
- [ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference](https://mlsanity.com/paper/2410.21465/) (2024)
- [SparQ Attention: Bandwidth-Efficient LLM Inference](https://mlsanity.com/paper/2312.04985/) (2023)
- [No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization](https://mlsanity.com/paper/2402.18096/) (2024)
- [PQCache: Product Quantization-based KVCache for Long Context LLM Inference](https://mlsanity.com/paper/2407.12820/) (2024)
- [Loki: Low-rank Keys for Efficient Sparse Attention](https://mlsanity.com/paper/2406.02542/) (2024)
- [Eigen Attention: Attention in Low-Rank Space for KV Cache Compression](https://mlsanity.com/paper/2408.05646/) (2024)
- [ThinK: Thinner Key Cache by Query-Driven Pruning](https://mlsanity.com/paper/2407.21018/) (2024)
- [LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy](https://mlsanity.com/paper/2410.03111/) (2024)
- [KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference](https://mlsanity.com/paper/2502.04420/) (2025)
- [Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models](https://mlsanity.com/paper/2406.08903/) (2024)
- [Optimised Grouped-Query Attention Mechanism for Transformers](https://mlsanity.com/paper/2406.14963/) (2024)
- [Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression](https://mlsanity.com/paper/2405.12591/) (2024)
- [Align Attention Heads Before Merging Them: An Effective Way for Converting MHA to GQA](https://mlsanity.com/paper/2412.20677/) (2024)

## Built on by (most cited first, 6 of 6)

- [KV Cache Transform Coding for Compact Storage in LLM Inference](https://mlsanity.com/paper/2511.01815/) (2025)
- [eOptShrinkQ: Near-Lossless KV Cache Compression Through Optimal Spectral Denoising and Quantization](https://mlsanity.com/paper/2605.02905/) (2026)
- [LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding](https://mlsanity.com/paper/2607.24555/) (2026)
- [KV-COBRA: KV Cache Compression via Co-Optimized Bit-Rank Allocation](https://mlsanity.com/paper/2609.24298/) (2026)
- [SPQ: An Ensemble Technique for Large Language Model Compression](https://mlsanity.com/paper/2602.18420/) (2026)
- [TaRA: Training-Aware Low-Rank Adaptation Initialization](https://mlsanity.com/paper/2609.02639/) (2026)

## Abstract (arXiv)

For the efficient inference of Large Language Models (LLMs), the effective compression of key-value (KV) cache is essential. Three main types of KV cache compression techniques, namely sparsity, channel compression, and quantization, have been identified. This study presents SVDq, a Singular Value Decomposition (SVD) - based mixed precision quantization method for K cache. Initially, K cache is transformed into latent channels using SVD basis representations. Since the values in latent channels decay rapidly and become negligible after only a few latent channels, our method then incorporates importance-aware quantization and compression for latent channels. This enables the effective allocation of higher precision to more significant channels. Theoretically, we prove that SVDq results in quantization errors (x0.1 or even lower) that are much lower than those of per-channel key quantization in the original space. Our findings based on RULER and LongBench benchmarks demonstrate that SVDq can achieve an equivalent key cache precision as low as 1.25-bit. When combined with key sparsity, it can reach a key compression ratio of up to 410x for attention computation, all while maintaining comparable model performance. Notably, our method is nearly lossless for LongBench datasets. This indicates that SVDq enables high-precision low-bit quantization, providing a more efficient solution for KV cache compression in LLMs.