mlsanityData to 7 Oct 2026

KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference

arXiv 2502.04420 ↗First posted 6 Feb 2025cs.LG cs.AI cs.CLcited by 36

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

KV cache quantization can improve Large Language Models (LLMs) inference throughput and latency in long contexts and large batch-size scenarios while preserving LLMs effectiveness. However, current methods have three unsolved issues: overlooking layer-wise sensitivity to KV cache quantization, high overhead of online fine-grained decision-making, and low flexibility to different LLMs and constraints. Therefore, we theoretically analyze the inherent correlation of layer-wise transformer attention patterns to KV cache quantization errors and study why key cache is generally more important than value cache for quantization error reduction. We further propose a simple yet effective framework KVTuner to adaptively search for the optimal hardware-friendly layer-wise KV quantization precision pairs for coarse-grained KV cache with multi-objective optimization and directly utilize the offline searched configurations during online inference. To reduce the computational cost of offline calibration, we utilize the intra-layer KV precision pair pruning and inter-layer clustering to reduce the search space. Experimental results show that we can achieve nearly lossless 3.25-bit mixed precision KV cache quantization for LLMs like Llama-3.1-8B-Instruct and 4.0-bit for sensitive models like Qwen2.5-7B-Instruct on mathematical reasoning tasks. The maximum inference throughput can be improved by 21.25% compared with KIVI-KV8 quantization over various context lengths. Our code and searched configurations are available at https://github.com/cmd2001/KVTuner.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 61 of its 69 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

61 of 69 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inference1 Jul
  2. 2024SpinQuant: LLM quantization with learned rotations26 May
  3. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification23 May
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  5. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache7 Mar
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 54

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023Mistral 7Bcited by 3,065
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  12. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
38 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  11. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  13. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  14. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  15. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  16. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  17. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  18. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  19. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  20. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  21. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  22. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  23. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  24. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  25. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  26. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  27. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  28. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  29. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  30. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  31. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  32. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  33. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  34. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  35. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  36. 2024IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intactcited by 52
  37. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  38. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 43

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 27.

36 papers.

Sort:

By area:

  1. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  2. 2025Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsitycited by 15
  3. 2025Quantize What Counts: More for Keys, Less for Valuescited by 9
  4. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studycited by 9
  5. 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boostcited by 8
  6. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  7. 2025SVDq: 1.25-bit and 410x Key Cache Compression for LLM Attentioncited by 6
  8. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  9. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4
  10. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inferencecited by 4
  11. 2025Towards Efficient Agents: A Co-Design of Inference Architecture and Systemcited by 4
  12. 2026Why Attention Patterns Exist: A Unifying Temporal Perspective Analysiscited by 4
24 more
  1. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
  2. 2026Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoningcited by 3
  3. 2026Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifoldcited by 3
  4. 2026A Simple Plug-in for Improving Eviction-Based KV Cache Compressioncited by 3
  5. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMscited by 2
  6. 2026Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formatscited by 2
  7. 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theorycited by 2
  8. 2026Search Your Block Floating Point Scales!cited by 2
  9. 2025A Unified Sparse Attention via Multi-Granularity Compressioncited by 1
  10. 2026PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantizationcited by 1
  11. 2026Detecting a Route Flip Is Easier Than Knowing Whether to Fix It: Causal Route-Mediated Damage in Quantized Mixture-of-Expertscited by 1
  12. 2026Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinementcited by 1
  13. 2026CompKV: Compensation-Aware KV Selection for Long-Context LLM Inferencecited by 1
  14. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  15. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantizationcited by 0
  16. 2026Minimal-Intervention KV Retention via Set-Conditioned Diversitycited by 0
  17. 2026Hurwitz Quaternion Multiplicative Quantization for KV Cache Compressioncited by 0
  18. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attentioncited by 0
  19. 2026HyQuant: Hybrid-Precision Quantization for LLM Attentioncited by 0
  20. 2026Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generationcited by 0
  21. 2026Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptationcited by 0
  22. 2026Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Servingcited by 0
  23. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 0
  24. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  2. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
  3. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  4. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
  5. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  6. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  7. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference
  8. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget