mlsanityData to 14 Oct 2026

NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

arXiv 2505.18231 ↗First posted 23 May 2025cs.LG cs.AIcited by 7

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large Language Model (LLM) inference is typically memory-intensive, especially when processing large batch sizes and long sequences, due to the large size of key-value (KV) cache. Vector Quantization (VQ) is recently adopted to alleviate this issue, but we find that the existing approach is susceptible to distribution shift due to its reliance on calibration datasets. To address this limitation, we introduce NSNQuant, a calibration-free Vector Quantization (VQ) technique designed for low-bit compression of the KV cache. By applying a three-step transformation-1) a token-wise normalization (Normalize), 2) a channel-wise centering (Shift), and 3) a second token-wise normalization (Normalize)-with Hadamard transform, NSNQuant effectively aligns the token distribution with the standard normal distribution. This alignment enables robust, calibration-free vector quantization using a single reusable codebook. Extensive experiments show that NSNQuant consistently outperforms prior methods in both 1-bit and 2-bit settings, offering strong generalization and up to 3× throughput gain over full-precision baselines. Code is available at https://github.com/DHdroid/NSNQuant.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 45 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

36 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024SpinQuant: LLM quantization with learned rotations26 May
  2. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification23 May
  3. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization7 May
  4. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
18 more
  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  5. 2018CoQA: A Conversational Question Answering Challengecited by 683
  6. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  7. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  8. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  9. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  10. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  11. 2025OstQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fittingcited by 97
  12. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 64
  13. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 58
  14. 2024VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Modelscited by 57
  15. 2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
  16. 2025BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMscited by 18
  17. 2023Benchmarking the Reliability of Post-training Quantization: a Particular Focus on Worst-case Performancecited by 9
  18. 2021SDR: Efficient Neural Re-ranking using Succinct Document Representationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  2. 2026Output-Aware Rotation for INT2 KV-Cache Quantizationcited by 2
  3. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsettingcited by 1
  4. 2026Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantizationcited by 0
  5. 2026Llama-Mobile: Efficient 2.7-Bit Quantization of VLMscited by 0
  6. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0
  7. 2026Tailoring the Quantization Space for 1-Bit KV Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  2. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  3. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
  4. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention
  5. 2026VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
  6. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
  7. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
  8. 2026E8 Lattice Key Quantization for Low-Bit KV-Cache Compression