mlsanityData to 7 Oct 2026

Accurate KV Cache Quantization with Outlier Tokens Tracing

arXiv 2505.10938 ↗First posted 16 May 2025cs.CLcited by 15

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The impressive capabilities of Large Language Models (LLMs) come at the cost of substantial computational resources during deployment. While KV Cache can significantly reduce recomputation during inference, it also introduces additional memory overhead. KV Cache quantization presents a promising solution, striking a good balance between memory usage and accuracy. Previous research has shown that the Keys are distributed by channel, while the Values are distributed by token. Consequently, the common practice is to apply channel-wise quantization to the Keys and token-wise quantization to the Values. However, our further investigation reveals that a small subset of unusual tokens exhibit unique characteristics that deviate from this pattern, which can substantially impact quantization accuracy. To address this, we develop a simple yet effective method to identify these tokens accurately during the decoding process and exclude them from quantization as outlier tokens, significantly improving overall accuracy. Extensive experiments show that our method achieves significant accuracy improvements under 2-bit quantization and can deliver a 6.4 times reduction in memory usage and a 2.3 times increase in throughput.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 48 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

42 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification23 May
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  3. 2024Accurate Block Quantization in LLMs with Outliers29 Mar
  4. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache7 Mar
  5. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
24 more
  1. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  13. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  14. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  15. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  16. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  17. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  18. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  19. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  20. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 71
  21. 2024With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generationcited by 36
  22. 2024Effectively Compress KV Heads for LLMcited by 30
  23. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 20
  24. 2024RelayAttention for Efficient Large Language Model Serving with Long System Promptscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 11.

15 papers.

Sort:

By area:

  1. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  2. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  3. 2025LOOM-Scope: a comprehensive and efficient LOng-cOntext Model evaluation frameworkcited by 5
  4. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  5. 2025Survey of Specialized Large Language Modelcited by 3
  6. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  7. 2026AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systemscited by 2
  8. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Managementcited by 1
  9. 2026RoPE-Aware Bit Allocation for KV-Cache Quantizationcited by 1
  10. 2026SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Codingcited by 1
  11. 2025PatternKV: Flattening KV Representation Expands Quantization Headroomcited by 0
  12. 2026LBLLM: Lightweight Binarization of Large Language Models via Three-Stage Distillationcited by 0
3 more
  1. 2026Hurwitz Quaternion Multiplicative Quantization for KV Cache Compressioncited by 0
  2. 2026OffQ: Taming Structured Outliers in LLM Quantization by Offsettingcited by 0
  3. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  2. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  3. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  4. 2024Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression
  5. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  6. 2025TaDA: Training-free recipe for Decoding with Adaptive KV Cache Compression and Mean-centering
  7. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  8. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction