mlsanityData to 7 Oct 2026

Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Caches

arXiv 2610.09827 ↗First posted 7 Oct 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Long inputs and extended generation increase the storage and access costs of the key-value (KV) cache. Low-bit quantization reduces storage and memory traffic, while query-channel pruning can further reduce key-cache reads. Rotation-based quantization redistributes the energy of key outliers across channels. To maintain computational invariance, the same orthogonal transform must be applied to queries, preserving query-key dot products. However, this rotation can disperse query energy, weakening the separation between a few large components to retain and many small ones to prune. We introduce Dual-QK, which uses paired non-orthogonal query and key transforms to address this conflict. Using calibrated query and key statistics, Dual-QK combines partial key whitening with a query-aligned basis to balance key scales for INT2 quantization and concentrate query energy for dynamic channel pruning. Channel-0 protection and bucket-relative RoPE support low-bit accuracy over long contexts. Experiments on four models across five generative benchmarks and long-context retrieval tasks show improved accuracy over OSCAR on most tasks at 40% query-channel sparsity. At a 128K context, Dual-QK provides 6.8× KV-cache compression and an estimated 8.3× reduction in KV read volume relative to unpruned BF16. Under the evaluated configurations, our SGLang implementation achieves up to 3.75× the decoding throughput of unpruned BF16.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 13 of its 47 references that are in the map. First posted on arXiv 7 Oct 2026.

Builds on

13 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms4 Aug
  2. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization18 May
  3. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving21 Apr
  4. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models10 May

Other references · 9

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  6. 2026Ministral 3cited by 88
  7. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  8. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  9. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Read What Matters: Query-Adaptive Quantization for KV Caches
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
  3. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Query
  4. 2025Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
  5. 2025PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration
  6. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
  7. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
  8. 2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantization