mlsanityData to 7 Oct 2026

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

arXiv 2402.02750 ↗First posted 5 Feb 2024cs.CL cs.LG cs.PFcited by 604

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Efficiently serving large language models (LLMs) requires batching of many requests to reduce the cost per request. Yet, with larger batch sizes and longer context lengths, the key-value (KV) cache, which stores attention keys and values to avoid re-computations, significantly increases memory demands and becomes the new bottleneck in speed and memory usage. Additionally, the loading of the KV cache causes the computational core to be idle, which limits the inference speed. A straightforward and effective solution to reduce KV cache size is quantization, which decreases the total bytes taken by KV cache. However, there is a lack of in-depth studies that explore the element distribution of KV cache to understand the hardness and limitation of KV cache quantization. To fill the gap, we conducted a comprehensive study on the element distribution in KV cache of popular LLMs. Our findings indicate that the key cache should be quantized per-channel, i.e., group elements along the channel dimension and quantize them together. In contrast, the value cache should be quantized per-token. From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using 2.6× less peak memory (including model weight). This reduction in memory usage enables up to 4× larger batch size, bringing 2.35× ~ 3.47× throughput on real LLM inference workload. The source code is available at https://github.com/jy-yuan/KIVI.

Key facts. Cited by 604 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 33 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

24 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Mistral 7Bcited by 3,065
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
10 more
  1. 2022Galactica: A Large Language Model for Sciencecited by 736
  2. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  5. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  7. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  8. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  9. 2023Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matchingcited by 30
  10. 2023Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Promptcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 95; 2025: 185; 2026: 324.

604 papers.

Sort:

By area:

  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  3. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  5. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  6. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  8. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  11. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  12. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
38 more
  1. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  3. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  5. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  6. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  7. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  8. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  9. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  10. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  11. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  12. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  13. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  14. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  15. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  16. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  17. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  18. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  19. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  20. 2024Evaluating Quantized Large Language Modelscited by 75
  21. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  22. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  23. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compressioncited by 65
  24. 2024Exploiting LLM Quantizationcited by 61
  25. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  26. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 59
  27. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  28. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  29. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
  30. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  31. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  32. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
  33. 2025KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environmentscited by 49
  34. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  35. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
  36. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  37. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  38. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44

50 most cited of 604 All 604 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  2. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  3. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  4. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  5. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
  6. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  7. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
  8. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization