# Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion

arXiv: https://arxiv.org/abs/2605.26266 · HTML: https://mlsanity.com/paper/2605.26266/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 25 May 2026.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)
- Efficient LLM inference and serving / [Fast autoregressive visual generation](https://mlsanity.com/inference/fast-autoregressive-visual-generation/)

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [Flex Attention: A Programming Model for Generating Optimized Attention Kernels](https://mlsanity.com/paper/2412.05496/) (2024)
- [WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling](https://mlsanity.com/paper/2512.14614/) (2025)
- [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://mlsanity.com/paper/2504.19874/) (2025)
- [SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantization](https://mlsanity.com/paper/2411.10958/) (2024)
- [Context Forcing: Consistent Autoregressive Video Generation with Long Context](https://mlsanity.com/paper/2602.06028/) (2026)
- [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2411.13503/) (2024)
- [Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization](https://mlsanity.com/paper/2602.02958/) (2026)
- [AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations](https://mlsanity.com/paper/2410.13212/) (2024)
- [Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention](https://mlsanity.com/paper/2602.01801/) (2026)
- [Softmax Bias Correction for Quantized Generative Models](https://mlsanity.com/paper/2309.01729/) (2023)
- [Timestep-Aware Correction for Quantized Diffusion Models](https://mlsanity.com/paper/2407.03917/) (2024)
- [Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attention](https://mlsanity.com/paper/2602.04789/) (2026)
- [KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction](https://mlsanity.com/paper/2510.05373/) (2025)
- [KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference](https://mlsanity.com/paper/2512.01953/) (2025)

## Built on by (most cited first, 2 of 2)

- [QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for World Models and Video Generation](https://mlsanity.com/paper/2609.26425/) (2026)
- [CORAL-KV: Compact Cross-Layer Prediction and Residual Coding for Autoregressive Video KV Caches](https://mlsanity.com/paper/2610.12514/) (2026)

## Abstract (arXiv)

Chunk-wise autoregressive video diffusion models rely on a KV cache of previously generated chunks to avoid redundant computation, but this cache quickly becomes a memory bottleneck as videos grow longer. Methods that quantize the KV cache to low bitwidths reduce memory pressure but degrade video quality. We show that a key driver of this degradation is a systematic bias in attention weights: due to the convexity of the exponential in softmax attention, quantization noise inflates the contribution of cached keys, a phenomenon we call the Jensen bias. This effect causes quantized keys to steal attention mass from the unquantized current chunk. We derive a per-attention-score correction that removes this bias in expectation, computed on the fly from the quantization step sizes of the cached keys and the query norm. Using a second-order Taylor approximation, the additional computational overhead is negligible, and no additional memory is needed alongside the cache. Evaluated on MAGI-1, SkyReels-V2, and HY-WorldPlay at INT2 quantization, our correction recovers most of the quality lost to aggressive quantization, reaching near-BF16 video quality, and can outperform INT4 quantization while using 50% less memory.