mlsanityData to 7 Oct 2026

Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression

arXiv 2405.12591 ↗First posted 21 May 2024cs.CLcited by 7

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Key-value~(KV) caching is an important technique to accelerate the inference of large language models~(LLMs), but incurs significant memory overhead. To compress the size of KV cache, existing methods often compromise precision or require extra data for calibration, limiting their practicality in LLM deployment. In this paper, we introduce DecoQuant, a novel data-free low-bit quantization technique based on tensor decomposition methods, to effectively compress KV cache. Our core idea is to adjust the outlier distribution of the original matrix by performing tensor decomposition, so that the quantization difficulties are migrated from the matrix to decomposed local tensors. Specially, we find that outliers mainly concentrate on small local tensors, while large tensors tend to have a narrower value range. Based on this finding, we propose to apply low-bit quantization to the large tensor, while maintaining high-precision representation for the small tensor. Furthermore, we utilize the proposed quantization method to compress the KV cache of LLMs to accelerate the inference and develop an efficient dequantization kernel tailored specifically for DecoQuant. Through extensive experiments, DecoQuant demonstrates remarkable efficiency gains, showcasing up to a ~75% reduction in memory footprint while maintaining comparable generation quality.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 33 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

17 of 33 references.

References · 17

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  10. 2023Adapting Language Models to Compress Contextscited by 350
  11. 2015Tensorizing Neural Networkscited by 246
  12. 2017Introduction to Tensor Decompositions and their Applications in Machine Learningcited by 45
5 more
  1. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  2. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 28
  3. 2019Compressing deep neural networks by matrix product operatorscited by 27
  4. 2020A Model Compression Method with Matrix Product Operators for Speech Enhancementcited by 9
  5. 2020Compressing LSTM Networks by Matrix Product Operatorscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQLcited by 26
  3. 2024Lossless KV Cache Compression to 2%cited by 7
  4. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
  5. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
  6. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  7. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Managementcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
  2. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  3. 2025Accurate KV Cache Quantization with Outlier Tokens Tracing
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  5. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
  6. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
  7. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  8. 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering