mlsanityData to 14 Oct 2026

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

arXiv 2605.06675 ↗First posted 22 Apr 2026cs.LG cs.CL cs.ITcited by 3

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large language models cache all previously computed key-value (KV) pairs during generation, and this KV cache grows linearly with sequence length, making it a primary memory bottleneck for serving. Quantizing the KV cache to fewer bits reduces this cost, yet all current quantizers assign the same bit-width to every attention head, ignoring the large variation in head importance. A natural idea is to allocate more bits to important heads and fewer to the rest. We show, however, that such mixed-precision allocation has a hidden pitfall: each quantizer follows a different distortion curve D(b)=alpha*beta^{-b}, and the decay rate beta varies from 3.6 to 5.3 across quantizer designs. Applying one quantizer's distortion model to another inverts the allocation order and makes performance worse than uniform quantization. We call this failure mode distortion model mismatch and propose RateQuant to resolve it. RateQuant fits a per-quantizer distortion model from a small calibration set, then solves the resulting bit-allocation problem in closed form via reverse waterfilling from rate-distortion theory. On Qwen3-8B at 2.5 average bits, calibrated RateQuant reduces KIVI's perplexity from 49.3 to 14.9 (70% reduction) and improves QuaRot by 6.6 PPL. The entire calibration takes 1.6 s on a single GPU and adds zero overhead at inference time.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 49 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

37 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning22 Dec
  2. 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost23 Nov
  3. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs24 May
  4. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache18 May
  5. 2025Quantize What Counts: More for Keys, Less for Values20 Feb
  6. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference6 Feb
  7. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
16 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  4. 2022Efficiently Scaling Transformer Inferencecited by 539
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  6. 2021A White Paper on Neural Network Quantizationcited by 472
  7. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  8. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  9. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
  10. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  11. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  12. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  13. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  14. 2025Radio: Rate-Distortion Optimization for Large Language Model Compressioncited by 5
  15. 2025BAQ: Efficient Bit Allocation Quantization for Large Language Modelscited by 2
  16. 2025CoKV: Optimizing KV Cache Allocation via Cooperative Gamecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  2. 2026HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantizationcited by 0
  3. 2026DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
  2. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  3. 2025XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
  4. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  5. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations
  6. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
  7. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
  8. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression