mlsanityData to 14 Oct 2026

MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization

arXiv 2504.03661 ↗First posted 12 Mar 2025cs.DCcited by 3

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Large language models (LLMs) are increasingly utilized for complex tasks requiring longer context lengths, with some models supporting up to 128K or 1M tokens. This trend, however, presents significant challenges in inference speed and memory management. Quantization emerges as a promising approach to address the widening gap between LLM size and memory capacity. However, traditional quantization schemes often yield suboptimal compression results for KV caches due to two key factors: i) On-the-fly quantization and de-quantization, causing significant performance overhead; ii) Prevalence of outliers in KV values, challenging low-bitwidth uniform quantization. To this end, we propose MILLION, a novel quantization framework achieving low-bitwidth KV cache through product quantization. First, we conduct a thorough analysis of KV cache distribution, revealing the limitations of existing quantization schemes. Second, we introduce a non-uniform quantization algorithm based on product quantization, which efficiently compresses data while preserving accuracy. Third, we develop a high-performance GPU inference framework with efficient attention kernel and pipeline design for MILLION that leverages sparse computation and asynchronous quantization, significantly enhancing inference speed. Comprehensive evaluation results demonstrate that MILLION can achieve 4 bits quantization with trivial perplexity and accuracy loss, and achieve 2.09x end-to-end performance gains at 32K context length. Code is released at https://github.com/ZongwuWang/MILLION.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 34 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

27 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan

Other references · 23

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2020Longformer: The Long-Document Transformercited by 3,413
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  9. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  10. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
11 more
  1. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  2. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  3. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  4. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  5. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  6. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  8. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 101
  9. 2022Results of the NeurIPS'21 Challenge on Billion-Scale Approximate Nearest Neighbor Searchcited by 45
  10. 2024CHAI: Clustered Head Attention for Efficient LLM Inferencecited by 13
  11. 2024CORM: Cache Optimization with Recent Message for Large Language Model Inferencecited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantizationcited by 4
  2. 2025TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inferencecited by 1
  3. 2026HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architecturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference
  2. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inference
  3. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  4. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  5. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  6. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
  7. 2026SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
  8. 2024XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference