WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms
On the map: Efficient LLM inference and serving → KV cache quantization
Abstract (arXiv)
KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck, we introduce WUSH-KV for low-bit KV-cache quantization. It adapts WUSH, which constructs a data-aware transform from the second-order statistics of both factors in a matrix product to reduce quantization error. WUSH-KV uses calibration data to construct separate key and value transforms, with the value transform folded into the model weights and the key transform applied after RoPE. The transforms can be paired with clipped quantizers. For one such quantizer, QuEST INT, we show that, under mild assumptions, the WUSH transform is near-optimal. With this quantizer, WUSH-KV reduces layerwise reconstruction error and achieves the lowest end-to-end perplexity among other tested transforms. For end-to-end evaluation, we integrate WUSH-KV into SGLang using OSCAR-style percentile-clipped affine quantization. At 2-bit, WUSH-KV performs comparably to or outperforms the OSCAR transform across all evaluated models and downstream tasks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 20 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
19 of 20 references.
Closest ancestors · 5
Same direction, newest first
- 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization18 May
- 2024SpinQuant: LLM quantization with learned rotations26 May
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
- 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
Other references · 14
- 2023Let's Verify Step by Stepcited by 3,598
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
- 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
- 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
- 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 85
- 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
- 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 21
2 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026KV Cache Compression Through the Lens of Transform Coding
- 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
- 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead
- 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
- 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
- 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
- 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
- 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference