mlsanityData to 14 Oct 2026

ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification

arXiv 2405.14256 ↗First posted 23 May 2024cs.LG cs.AIcited by 93

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

KV cache stores key and value states from previous tokens to avoid re-computation, yet it demands substantial storage space, especially for long sequences. Adaptive KV cache compression seeks to discern the saliency of tokens, preserving vital information while aggressively compressing those of less importance. However, previous methods of this approach exhibit significant performance degradation at high compression ratios due to inaccuracies in identifying salient tokens. In this paper, we present ZipCache, an accurate and efficient KV cache quantization method for LLMs. First, we construct a strong baseline for quantizing KV cache. Through the proposed channel-separable tokenwise quantization scheme, the memory overhead of quantization parameters are substantially reduced compared to fine-grained groupwise quantization. To enhance the compression ratio, we propose normalized attention score as an effective metric for identifying salient tokens by considering the lower triangle characteristics of the attention matrix. Moreover, we develop an efficient approximation method that decouples the saliency metric from full attention scores, enabling compatibility with fast attention implementations like FlashAttention. Extensive experiments demonstrate that ZipCache achieves superior compression ratios, fast generation speed and minimal performance losses compared with previous KV cache compression methods. For instance, when evaluating Mistral-7B model on GSM8k dataset, ZipCache is capable of compressing the KV cache by 4.98×, with only a 0.38% drop in accuracy. In terms of efficiency, ZipCache also showcases a 37.3% reduction in prefill-phase latency, a 56.9% reduction in decoding-phase latency, and a 19.8% reduction in GPU memory usage when evaluating LLaMA3-8B model with a input length of 4096.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 45 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

34 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2023Mistral 7Bcited by 3,164
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
21 more
  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  6. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  8. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  10. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  11. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  13. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 170
  14. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  15. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  16. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  17. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  18. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 64
  19. 2022Token Dropping for Efficient BERT Pretrainingcited by 42
  20. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22
  21. 2021Investigating Math Word Problems using Pretrained Multilingual Language Modelscited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 32; 2026: 46.

93 papers.

Sort:

By area:

  1. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  2. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  4. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 96
  5. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 46
  6. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 44
  7. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 43
  8. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 40
  9. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  10. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 36
  11. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  12. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
38 more
  1. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 24
  2. 2026Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantizationcited by 23
  3. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  4. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 18
  5. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  6. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  7. 2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
  8. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 15
  9. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  10. 2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Querycited by 14
  11. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 13
  12. 2024Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Mannercited by 11
  13. 2025MEPIC: Memory Efficient Position Independent Caching for LLM Servingcited by 11
  14. 2025PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Accelerationcited by 10
  15. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimizationcited by 10
  16. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMindcited by 10
  17. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 10
  18. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 10
  19. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMscited by 9
  20. 2025SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layerscited by 9
  21. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 9
  22. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 9
  23. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 8
  24. 2024Lossless KV Cache Compression to 2%cited by 7
  25. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  26. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cachecited by 7
  27. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  28. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
  29. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  30. 2026IndexMem: Learned KV-Cache Eviction with Latent Memory for Long-Context LLM Inferencecited by 7
  31. 2025Curse of High Dimensionality Issue in Transformer for Long-context Modelingcited by 5
  32. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Servingcited by 5
  33. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  34. 2024AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantizationcited by 4
  35. 2025Quantize-Sample-and-Verify: LLM Acceleration via Adaptive Edge-Cloud Speculative Decodingcited by 4
  36. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
  37. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantizationcited by 4
  38. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for World Models and Video Generation
  2. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  3. 2026KV Cache Compression Through the Lens of Transform Coding
  4. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  5. 2025AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models
  6. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
  8. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization