mlsanityData to 7 Oct 2026

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

arXiv 2401.18079 ↗First posted 31 Jan 2024cs.LGcited by 571

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

LLMs are seeing growing use for applications which require large context windows, and with these large context windows KV cache activations surface as the dominant contributor to memory consumption during inference. Quantization is a promising approach for compressing KV cache activations; however, existing solutions fail to represent activations accurately in sub-4-bit precision. Our work, KVQuant, facilitates low precision KV cache quantization by incorporating several novel methods: (i) Per-Channel Key Quantization, where we adjust the dimension along which we quantize the Key activations to better match the distribution; (ii) Pre-RoPE Key Quantization, where we quantize Key activations before the rotary positional embedding to mitigate its impact on quantization; (iii) Non-Uniform KV Cache Quantization, where we derive per-layer sensitivity-weighted non-uniform datatypes that better represent the distributions; and (iv) Per-Vector Dense-and-Sparse Quantization, where we isolate outliers separately for each vector to minimize skews in quantization ranges. By applying our method to the LLaMA, Llama-2, Llama-3, and Mistral models, we achieve < 0.1 perplexity degradation with 3-bit quantization on both Wikitext-2 and C4, outperforming existing approaches. Our method enables serving LLaMA-7B with a context length of up to 1 million on a single A100-80GB GPU and up to 10 million on an 8-GPU system. We develop custom CUDA kernels for KVQuant, showing that we can achieve up to ~1.7x speedups, compared to baseline fp16 matrix-vector multiplications, for the LLaMA-7B model.

Key facts. Cited by 571 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 51 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

31 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 30

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023Mistral 7Bcited by 3,065
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
18 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  5. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  6. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  7. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  8. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  9. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  10. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  12. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  13. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  14. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  15. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  16. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  17. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  18. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 105; 2025: 186; 2026: 280.

571 papers.

Sort:

By area:

  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  4. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  5. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  6. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  9. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  11. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  12. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
38 more
  1. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  3. 2025MemOS: A Memory OS for AI Systemcited by 129
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  5. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  6. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  7. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  8. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  9. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  10. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  11. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  12. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  13. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  14. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  15. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  16. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  17. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  18. 2024GPTVQ: The Blessing of Dimensionality for LLM Quantizationcited by 78
  19. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  20. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  21. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 71
  22. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  23. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  24. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  25. 2024CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoningcited by 64
  26. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  27. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  28. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  29. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  30. 2024Base of RoPE Bounds Context Lengthcited by 49
  31. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  32. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  33. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
  34. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  35. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  36. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  37. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  38. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 43

50 most cited of 571 All 571 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  2. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  3. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
  4. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
  5. 2025PolarQuant: Quantizing KV Caches with Polar Transformation
  6. 2025LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation
  7. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  8. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference