mlsanityData to 14 Oct 2026

RoPE-Aware Bit Allocation for KV-Cache Quantization

arXiv 2606.24033 ↗First posted 23 Jun 2026cs.LG cs.CLcited by 1

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Existing low-bit KV-cache quantizers often treat each cached key as a flat vector. Under RoPE, however, a key's contribution to a future attention logit decomposes into a position-dependent sum over two-dimensional frequency blocks. This makes key-cache quantization a block-wise bit-allocation problem: high-energy RoPE blocks are more sensitive to quantization error and should receive more bits. We introduce Block-GTQ, a RoPE-aware bit allocator for key-cache quantization built on TurboQuant-MSE(TQ-MSE). For each layer and KV head, Block-GTQ computes a label-free energy score for each RoPE block and greedily allocates integer bit widths by marginal gain. Under matched K/V bit budgets, Block-GTQ better preserves RoPE query-key logits on a ten-model diagnostic panel, cutting per-layer MAE by 32-80% at 2 and 3 b/dim K-only quantization and winning all 367/367 layer comparisons against uniform TQ-MSE. These fidelity gains translate to stronger downstream long-context retrieval, understanding, and reasoning. At K2V2 on Llama-3.1-8B-Instruct, Block-GTQ raises the six-task NIAH average from 70.6 to 97.4, and the LongBench-EN average from 36.87 to 53.31. On AIME 2024/2025 with DeepSeek-R1-Distill-Qwen-7B, without an fp16 recent-key buffer, Block-GTQ at K3V2 scores 51.7/37.5, close to fp16's 54.2/37.9, whereas uniform TQ-MSE collapses to 0.0/0.0. We further implement a packed-cache serving path. On a single H800 GPU with Qwen2.5-3B-Instruct, packed K3V3 achieves 3.24x KV-cache compression with fp16-comparable quality, runs 1.34x faster than fp16 FlashAttention2 at 128K context, reduces peak memory from 56.31 GB to 19.85 GB, and remains feasible at 256K and 512K where fp16 OOMs. Code is available at https://github.com/JIA-Lab-research/blockgtq.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 42 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

40 of 42 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning22 Dec
  2. 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost23 Nov
  3. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs6 Aug
  4. 2025CommVQ: Commutative Vector Quantization for KV Cache Compression23 Jun
  5. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs24 May
  6. 2025Accurate KV Cache Quantization with Outlier Tokens Tracing16 May
  7. 2025SQuat: Subspace-orthogonal KV Cache Quantization31 Mar
  8. 2025PolarQuant: Quantizing KV Caches with Polar Transformation4 Feb
  9. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations25 Jan
  10. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification23 May
  11. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization7 May
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
1 more
  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 27

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2022Efficiently Scaling Transformer Inferencecited by 539
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
15 more
  1. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  2. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  4. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  5. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  6. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  7. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  8. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  9. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 64
  10. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 24
  11. 2024SubGen: Token Generation in Sublinear Time and Memorycited by 21
  12. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 21
  13. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  14. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  15. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  2. 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization
  3. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations
  4. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization
  5. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
  6. 2026E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
  7. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms
  8. 2026Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation