mlsanityData to 7 Oct 2026

WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

arXiv 2607.28699 ↗First posted 30 Jul 2026cs.AR cs.AIcited by 3

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it is serving right now. We give it a provably sound runtime meter -- a "DTrace for KV quantization": a per-(layer, head, step) upper bound on the total variation between exact and compressed attention. The meter has two tiers: a deterministic band-norm-witness bound, sound for any cache-preserving black-box quantizer and for any query (adaptive-safe, worst-case Cauchy--Schwarz plus RoPE band-unitarity), and a tighter probabilistic certificate for a controlled subtractively-dithered INT8 quantizer under an explicit request-level failure budget (stated for non-adaptive queries; core theorems machine-checked in Lean 4). Three results. Observability: the meter enters SGLang through an env-guarded patch, and any scheme registered as one tensor function is measured in live serving. Repair: meter-driven gating -- risk-ranked where the witness is saturated, certified where it is informative -- empirically restores the quality floor at benchmark scale, e.g. raw-cast fp8 from 22.8 back to 79.7 on hard RULER tasks with the difference from uncompressed bounded at [+0.0,+0.8] by a paired test. Analysis: aggressive schemes survive on cross-layer error cancellation, not per-step fidelity -- in a 28-layer sweep, no single layer's pollution alone loses anything (0/28) -- and the certified int8 cache serves 1.88× more KV tokens at the same memory in SGLang. All artifacts, guards, and the Lean development are released at https://github.com/metask-ai/witcert-kv-certificates; every number regenerates from the shipped artifacts by one command.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 50 references that are in the map. First posted on arXiv 30 Jul 2026.

Builds on

34 of 50 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference30 Jun
  2. 2026RoPE-Aware Bit Allocation for KV-Cache Quantization23 Jun
  3. 2026KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks2 Jun
  4. 2026Runtime-Certified Bounded-Error Quantized Attention20 May
  5. 2025CommVQ: Commutative Vector Quantization for KV Cache Compression23 Jun
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 27

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  10. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
15 more
  1. 2024Massive Activations in Large Language Modelscited by 277
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  3. 2019The Lean mathematical librarycited by 127
  4. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  5. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  6. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  7. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  8. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
  9. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  10. 2025Streaming Attention Approximation via Discrepancy Theorycited by 9
  11. 2025KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelitycited by 9
  12. 2025EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projectioncited by 7
  13. 2025KV-Latent: Dimensional-level KV Cache Reduction with Frequency-aware Rotary Positional Embeddingcited by 5
  14. 2026The risk of KV cache compressioncited by 4
  15. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Runtime Observability for Heterogeneous Attention Memorycited by 1
  2. 2026Pricing the Risk of Runtime Compression: Anytime-Valid Admission and a Served-Output Law for Compressed Serving Statecited by 0
  3. 2026Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
  2. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  3. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  4. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  5. 2025Accurate KV Cache Quantization with Outlier Tokens Tracing
  6. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization
  7. 2026OmniKVQuant: KV Cache Quantization for Omni-LLMs
  8. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization