mlsanityData to 7 Oct 2026

MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache

arXiv 2411.18077 ↗First posted 27 Nov 2024cs.CL cs.LGcited by 6

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

How to efficiently serve LLMs in practice has become exceptionally challenging due to their prohibitive memory and computation requirements. In this study, we investigate optimizing the KV cache, whose memory footprint poses a critical bottleneck in LLM inference, especially when dealing with long context tasks. To tackle the challenge, we introduce MiniKV, a KV cache optimization method that simultaneously preserves long context task accuracy while significantly reducing KV cache size via a novel 2-bit layer-discriminative KV cache. More importantly, we develop specialized CUDA kernels to make MiniKV compatible with FlashAttention. Experiments on a wide range of long context tasks show that MiniKV effectively achieves 86% KV cache compression ratio while recovering over 98.5% of accuracy, outperforming state-of-the-art methods while achieving excellent measured system performance improvements.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 44 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

33 of 44 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  3. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  4. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  5. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  6. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  8. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization28 Feb
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
10 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  5. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  6. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  7. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  8. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  9. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  10. 2021Multi-head or Single-head? An Empirical Comparison for Transformer Trainingcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 4.

6 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boostcited by 8
  3. 2026OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantizationcited by 7
  4. 2026MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compressioncited by 1
  5. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  6. 2026Minimal-Intervention KV Retention via Set-Conditioned Diversitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
  2. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  3. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  4. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2025ReFreeKV: Towards Threshold-Free KV Cache Compression
  7. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  8. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference