HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
On the map: Efficient LLM inference and serving → KV cache quantization
Abstract (arXiv)
KV-cache quantizers usually optimize storage-space reconstruction, even though attention reads keys through logits and values through attention-weighted readout. We argue that persistent cache error should be measured in model-visible coordinates. For keys, the visible object is score error modulo constant shifts; this yields HeadQ, a key-side method that stores a low-rank residual side code in a calibration-learned query basis and applies it as an additive logit correction. For values, fixed-attention readout gives an A2-weighted token-distortion surrogate. Across six models, Fisher/score-space error predicts attention KL far better than raw key MSE; same-budget counterexamples, null-space interventions, query-PCA controls, and wrong-sign HeadQ falsify storage-MSE alternatives. Matched Pythia checkpoints localize the main anomaly to a small-model low-entropy route-flip boundary. In K-only WikiText-103 decode experiments with dense values, HeadQ removes roughly 84--94% of the excess perplexity on the strongest 2-bit rows; in an auxiliary full-KV 2-bit composition, HeadQ plus an A2 value policy improves all six models.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 24 references that are in the map. First posted on arXiv 5 May 2026.
Builds on
20 of 24 references.
Closest ancestors · 9
Same direction, newest first
- 2026AXELRAM: Quantize Once, Never Dequantize3 Apr
- 2025Quantize What Counts: More for Keys, Less for Values20 Feb
- 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference6 Feb
- 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations25 Jan
- 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations17 Oct
- 2024SpinQuant: LLM quantization with learned rotations26 May
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
Other references · 11
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2024Qwen2 Technical Reportcited by 2,076
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
- 2024TinyLlama: An Open-Source Small Language Modelcited by 604
- 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
- 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
- 2026LOOKAT: Lookup-Optimized Key-Attention for Memory-Efficient Transformerscited by 3
- 2026Exact Attention Sensitivity and the Geometry of Transformer Stabilitycited by 3
- 2026PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inferencecited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Read What Matters: Query-Adaptive Quantization for KV Caches
- 2026Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation
- 2026Runtime-Certified Bounded-Error Quantized Attention
- 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
- 2026KV Cache Compression Through the Lens of Transform Coding
- 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
- 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
- 2026ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression