mlsanityData to 7 Oct 2026

Runtime-Certified Bounded-Error Quantized Attention

arXiv 2605.20868 ↗First posted 20 May 2026cs.LG cs.AI cs.SYcited by 4

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

KV cache quantization reduces the memory cost of long-context LLM inference, but introduces approximation error that is typically validated only empirically. Existing systems rely on average-case robustness, with no mechanism to detect or recover from failures at runtime. We present a tiered KV cache architecture that enables runtime-certified attention: INT8 keys and INT4 values are stored in GPU memory, while FP16 originals are retained in system RAM for deterministic fallback. A two-term error decomposition yields per-head, per-step bounds on (i) attention distribution distortion from key quantization and (ii) value reconstruction error. These bounds are computed online and used to drive adaptive precision selection and a multi-stage fallback ladder, which guarantees recovery to the exact dense attention output when required. Across PG-19, NIAH, and RULER benchmarks on LLaMA~3.1-8B with contexts up to 128K, the system matches dense FP16 KV quality within noise for language modelling and retrieval tasks, while recovering catastrophic failures observed in naive INT8/INT4 baselines. Value-sensitive tasks at short context expose a controlled trade-off between compression and fidelity, which can be eliminated via tighter value tolerances or FP16-value fallback. The certification is local (per-head, per-step) and does not guarantee end-to-end model correctness, but ensures that each attention computation is either bounded relative to an FP16 reference or exactly recovered via fallback. This reframes KV cache quantization as a runtime-verified computation rather than a fixed approximation. The goal is not raw speedups, but enabling safe deployment of aggressive KV compression under strict quality constraints.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 37 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

22 of 37 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMs6 Apr
  2. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models26 Feb
  3. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization7 Oct
  4. 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering5 Jun
  5. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference30 Mar
  6. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inference1 Jul
  7. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead5 Jun
  8. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 12

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  6. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  7. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  8. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  9. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 12
  10. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inferencecited by 10
  11. 2025A Mathematical Theory of Top-k Sparse Attention via Total Variation Distancecited by 4
  12. 2026PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inferencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3
  2. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  3. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  4. 2026Pricing the Risk of Runtime Compression: Anytime-Valid Admission and a Served-Output Law for Compressed Serving Statecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  2. 2026Output-Aware Rotation for INT2 KV-Cache Quantization
  3. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  4. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
  5. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
  6. 2026Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation
  7. 2026RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
  8. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs