mlsanityData to 14 Oct 2026

CommVQ: Commutative Vector Quantization for KV Cache Compression

arXiv 2506.18879 ↗First posted 23 Jun 2025cs.CL cs.AIcited by 28

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large Language Models (LLMs) are increasingly used in applications requiring long context lengths, but the key-value (KV) cache often becomes a memory bottleneck on GPUs as context grows. To address this, we propose Commutative Vector Quantization (CommVQ) to significantly reduce memory usage for long-context LLM inference. We first introduce additive quantization with a lightweight encoder and codebook to compress the KV cache, which can be decoded via simple matrix multiplication. To further reduce computational costs during decoding, we design the codebook to be commutative with Rotary Position Embedding (RoPE) and train it using an Expectation-Maximization (EM) algorithm. This enables efficient integration of decoding into the self-attention mechanism. Our approach achieves high accuracy with additive quantization and low overhead via the RoPE-commutative codebook. Experiments on long-context benchmarks and GSM8K show that our method reduces FP16 KV cache size by 87.5% with 2-bit quantization, while outperforming state-of-the-art KV cache quantization methods. Notably, it enables 1-bit KV cache quantization with minimal accuracy loss, allowing a LLaMA-3.1 8B model to run with a 128K context length on a single RTX 4090 GPU. The source code is available at: https://github.com/UMass-Embodied-AGI/CommVQ.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 29 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

22 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Residual vector quantization for KV cache compression in large language model21 Oct
  2. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization7 May
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  9. 2023Mistral 7Bcited by 3,164
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
6 more
  1. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  2. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  4. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  6. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 25.

28 papers.

Sort:

By area:

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  2. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  3. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devicescited by 8
  4. 2025V-Rex: Real-Time Streaming Video LLM Acceleration via Dynamic KV Cache Retrievalcited by 5
  5. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Servingcited by 5
  6. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Samplingcited by 5
  7. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  8. 2026Memory for Large Language Modelscited by 4
  9. 2026WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantizationcited by 3
  10. 2026EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecturecited by 2
  11. 2026Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transformscited by 2
  12. 2026T-LoopFormer: Token-Level Elastic-Depth Looped Transformers for Latent Reasoning with Dynamic Routingcited by 2
16 more
  1. 2025Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoderscited by 1
  2. 2026Online Vector Quantized Attentioncited by 1
  3. 2026RoPE-Aware Bit Allocation for KV-Cache Quantizationcited by 1
  4. 2026AnchorKV: Anchor-Residual KV Cache Compressioncited by 1
  5. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Modelscited by 1
  6. 2026KV Cache Compression Through the Lens of Transform Codingcited by 1
  7. 2026QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for World Models and Video Generationcited by 1
  8. 2026GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMscited by 0
  9. 2026Hurwitz Quaternion Multiplicative Quantization for KV Cache Compressioncited by 0
  10. 2026When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisationcited by 0
  11. 2026A Target-Centric Survey of Quantization-Aware Trainingcited by 0
  12. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0
  13. 2026PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attentioncited by 0
  14. 2026RECAP: Calibrating Low-Bit Reasoning Models on the History They Readcited by 0
  15. 2026TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generationcited by 0
  16. 2026E8 Lattice Key Quantization for Low-Bit KV-Cache Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
  2. 2026VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
  3. 2026Tailoring the Quantization Space for 1-Bit KV Cache Compression
  4. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  5. 2026GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models
  6. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Models
  7. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
  8. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention