Shared Stopping Decisions Change Answers in HQQ Cache Quantization
Not on the map. Why?
Abstract (arXiv)
Language-model systems batch questions for throughput, but unrelated questions should not change a target's answer when its input and numerical execution are fixed. We study compression of the key and value cache, which stores attention representations reused during generation. With request-local groups, Transformers' Half-Quadratic Quantization (HQQ) backend updates compression parameters separately but uses a shared average error to decide when all updates stop. Replacing only the question batched with the target changes four-bit HQQ answers in 170/384 test comparisons across two models. Replaying the other execution's update counts reproduces its complete answer and cache fingerprints in every changed pair, in both directions. Computing the stopping mean in FP32 reduces cache differences but leaves answer changes. Native HQQ also changes confirmed numerical correctness in eight arithmetic pairs. Fixed iterations and request-local stopping remove observed companion dependence under matched controls. Request-local stopping remains sensitive to synthetic padding changes at the tensor level. Fixing the original iteration budget removes this decision path without tuning. Neither repair has an established quality advantage, and natural rebatching still changes answers. Request-independence audits must cover stopping decisions as well as quantization groups.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 6 of its 28 references that are in the map. First posted on arXiv 5 Oct 2026.
Builds on
6 of 28 references.
References · 6
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2025Qwen3 Technical Reportcited by 8,820
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
- 2024Phi-4 Technical Reportcited by 711
- 2026Quantamination: Dynamic Quantization Leaks Your Data Across the Batchcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Mask-Guided KV Cache Eviction in Block Diffusion Language Models
- 2026Small Frequency Corrections Can Change What Survives KV Cache Compression
- 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
- 2026Runtime-Certified Bounded-Error Quantized Attention
- 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads
- 2026Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
- 2025REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression
- 2025Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression