mlsanityData to 7 Oct 2026

QAQ: Quality Adaptive Quantization for LLM KV Cache

arXiv 2403.04643 ↗First posted 7 Mar 2024cs.CLcited by 66

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The emergence of LLMs has ignited a fresh surge of breakthroughs in NLP applications, particularly in domains such as question-answering systems and text generation. As the need for longer context grows, a significant bottleneck in model deployment emerges due to the linear expansion of the Key-Value (KV) cache with the context length. Existing methods primarily rely on various hypotheses, such as sorting the KV cache based on attention scores for replacement or eviction, to compress the KV cache and improve model throughput. However, heuristics used by these strategies may wrongly evict essential KV cache, which can significantly degrade model performance. In this paper, we propose QAQ, a Quality Adaptive Quantization scheme for the KV cache. We theoretically demonstrate that key cache and value cache exhibit distinct sensitivities to quantization, leading to the formulation of separate quantization strategies for their non-uniform quantization. Through the integration of dedicated outlier handling, as well as an improved attention-aware approach, QAQ achieves up to 10x the compression ratio of the KV cache size with a neglectable impact on model performance. QAQ significantly reduces the practical hurdles of deploying LLMs, opening up new possibilities for longer-context applications. The code is available at github.com/ClubieDong/KVCacheQuantization.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 21 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

15 of 21 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  10. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  11. 2022Efficiently Scaling Transformer Inferencecited by 529
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
3 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  3. 2023A Survey on Model Compression for Large Language Modelscited by 263

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 28; 2026: 17.

66 papers.

Sort:

By area:

  1. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  3. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  4. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  6. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  7. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
  8. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  9. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  10. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  11. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  12. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
38 more
  1. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  2. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  3. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  4. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  5. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 15
  6. 2025KV Cache Compression for Inference Efficiency in LLMs: A Reviewcited by 12
  7. 2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 10
  8. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  9. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 10
  10. 2025Streaming Attention Approximation via Discrepancy Theorycited by 9
  11. 2025Quantize What Counts: More for Keys, Less for Valuescited by 9
  12. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 9
  13. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9
  14. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 8
  15. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inferencecited by 8
  16. 2025PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Accelerationcited by 7
  17. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  18. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Servingcited by 7
  19. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
  20. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  21. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inferencecited by 5
  22. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  23. 2024AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantizationcited by 4
  24. 2024An Evolved Universal Transformer Memorycited by 4
  25. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4
  26. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantizationcited by 4
  27. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
  28. 2026Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generationcited by 4
  29. 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centeringcited by 3
  30. 2024Efficient LLM Inference with Kcachecited by 2
  31. 2024Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inferencecited by 2
  32. 2024Art and Science of Quantizing Large-Scale Models: A Comprehensive Overviewcited by 2
  33. 2026EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstructioncited by 2
  34. 2026OCTOPUS: Optimized KV Cache for Transformers via Octahedral Parametrization Under optimal Squared error quantizationcited by 2
  35. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  36. 2025iServe: An Intent-based Serving System for LLMscited by 1
  37. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  38. 2025CaliDrop: KV Cache Compression with Calibrationcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  2. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
  3. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
  4. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
  5. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  6. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  7. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache