mlsanityData to 14 Oct 2026

SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models

arXiv 2405.06219 ↗First posted 10 May 2024cs.LG cs.CLcited by 44

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large language models (LLMs) can now handle longer sequences of tokens, enabling complex tasks like book understanding and generating lengthy novels. However, the key-value (KV) cache required for LLMs consumes substantial memory as context length increasing, becoming the bottleneck for deployment. In this paper, we present a strategy called SKVQ, which stands for sliding-window KV cache quantization, to address the issue of extremely low bitwidth KV cache quantization. To achieve this, SKVQ rearranges the channels of the KV cache in order to improve the similarity of channels in quantization groups, and applies clipped dynamic quantization at the group level. Additionally, SKVQ ensures that the most recent window tokens in the KV cache are preserved with high precision. This helps maintain the accuracy of a small but important portion of the KV cache.SKVQ achieves high compression ratios while maintaining accuracy. Our evaluation on LLMs demonstrates that SKVQ surpasses previous quantization approaches, allowing for quantization of the KV cache to 2-bit keys and 1.5-bit values with minimal loss of accuracy. With SKVQ, it is possible to process context lengths of up to 1M on an 80GB memory GPU for a 7b model and up to 7 times faster decoding.

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 32 references that are in the map. First posted on arXiv 10 May 2024.

Builds on

26 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  3. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
11 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  3. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  4. 2019Revealing the Dark Secrets of BERTcited by 301
  5. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  6. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  7. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  9. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 116
  10. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  11. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 20; 2026: 14.

44 papers.

Sort:

By area:

  1. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  2. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  4. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  5. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  6. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  7. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  8. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 30
  9. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 24
  10. 2026Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantizationcited by 23
  11. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 22
  12. 2025M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Typecited by 16
32 more
  1. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 13
  2. 2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 11
  3. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generationcited by 11
  4. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 10
  5. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
  6. 2024BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inferencecited by 9
  7. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 9
  8. 2025AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Modelscited by 9
  9. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
  10. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 8
  11. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  12. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  13. 2025Compression Barriers for Autoregressive Transformerscited by 5
  14. 2025Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Servingcited by 5
  15. 2026SliderQuant: Accurate Post-Training Quantization for LLMscited by 5
  16. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  17. 2026InnerQ: Hardware-Aware Tuning-Free Quantization of KV Cache for Large Language Modelscited by 3
  18. 2024Art and Science of Quantizing Large-Scale Models: A Comprehensive Overviewcited by 2
  19. 2025Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decodingcited by 2
  20. 2026AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantizationcited by 2
  21. 2025SAIL: SRAM-Accelerated LLM Inference System with Lookup-Table-based GEMVcited by 1
  22. 2024QET: Enhancing Quantized LLM Parameters and KV cache Compression through Element Substitution and Residual Clusteringcited by 0
  23. 2025PatternKV: Flattening KV Representation Expands Quantization Headroomcited by 0
  24. 2025SHUFFLESPARSE: Learned Shuffles for Structured Sparse Networkscited by 0
  25. 2026Dual Dimensionality for Local and Global Attentioncited by 0
  26. 2026SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inferencecited by 0
  27. 2026Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptationcited by 0
  28. 2026QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Cachingcited by 0
  29. 2026Tailoring the Quantization Space for 1-Bit KV Cache Compressioncited by 0
  30. 2026Dual-QK: Sharp Queries and Flat Keys for Prunable 2-bit KV Cachescited by 0
  31. 2026KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantizationcited by 0
  32. 2026TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  2. 2026VQKV: High-Fidelity and High-Ratio Cache Compression via Vector-Quantization
  3. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
  4. 2026E8 Lattice Key Quantization for Low-Bit KV-Cache Compression
  5. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attention
  6. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference
  7. 2026High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
  8. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference