mlsanityData to 7 Oct 2026

High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

arXiv 2607.16248 ↗First posted 27 Jun 2026cs.LG cs.AIcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Long-context large language model inference relies on the KV cache to avoid redundant attention computation, but incurs high memory and bandwidth overheads. Low-bit KV-cache quantization reduces this cost, yet it severely degrade quality; particularly, one-bit quantization reduces accuracy from 84.2% to 47.8% on Llama-3.1-8B under RULER. Rather than common beliefs that absolute error of logits, we find that the root cause is structured local misranking, where the distribution of logits in top-K region is drifted. We thereby propose local distribution restoration, a new technique that detects steps with high local distribution risk from quantized-logit features and restores only the selected top-K candidate distribution before token selection. We implement DGAP to achieve local distribution restoration, with efficient risk detcetors and correctors. Expeirments show that on Llama-3.1-8B, DGAP recovers K1V1 RULER accuracy from 47.8% to 83.2% and reduces distribution drift from 0.38 to 0.14; across Llama, Mistral, and Qwen models, it preserves the persistent low-bit KV-cache footprint with modest decode overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 25 references that are in the map. First posted on arXiv 27 Jun 2026.

Builds on

21 of 25 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs24 May
  2. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations17 Oct
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
5 more
  1. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  2. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  3. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  4. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 20
  5. 2025DartQuant: Efficient Rotational Distribution Calibration for LLM Quantizationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
  2. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  3. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  4. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  5. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  6. 2026SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
  7. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models
  8. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression