mlsanityData to 14 Oct 2026

NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics

arXiv 2505.16210 ↗First posted 22 May 2025cs.LG cs.AI cs.CLcited by 3

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable proficiency across a wide range of tasks. However, LLMs often require larger batch sizes to enhance throughput or longer context lengths to meet task demands, which significantly increases the memory resource consumption of the Key-Value (KV) cache during inference, becoming a major bottleneck in LLM deployment. To address this issue, quantization is a common and straightforward approach. Currently, quantization methods for activations are limited to 8-bit, and quantization to even lower bits can lead to substantial accuracy drops. To further save space by quantizing the KV cache to even lower bits, we analyzed the element distribution of the KV cache and designed the NQKV algorithm. Since the elements within each block of the KV cache follow a normal distribution, NQKV employs per-block quantile quantization to achieve information-theoretically optimal quantization error. Without significantly compromising model output quality, NQKV enables the OPT model to perform inference with an 2x larger batch size or a 4x longer context length, and it improves throughput by 9.3x compared to when the KV cache is not used.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 38 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

24 of 38 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
12 more
  1. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  2. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  5. 2021A White Paper on Neural Network Quantizationcited by 472
  6. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  7. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  8. 2023A Survey on Model Compression for Large Language Modelscited by 269
  9. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  10. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  11. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  12. 2023Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
  2. 2026Gated Differentiable Working Memory for Long-Context Language Modelingcited by 6
  3. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transformscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  2. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  3. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
  4. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
  5. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  6. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache
  7. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  8. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache