mlsanityData to 7 Oct 2026

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

arXiv 2405.04532 ↗First posted 7 May 2024cs.CL cs.AI cs.LGcited by 200

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Quantization can accelerate large language model (LLM) inference. Going beyond INT8 quantization, the research community is actively exploring even lower precision, such as INT4. Nonetheless, state-of-the-art INT4 quantization techniques only accelerate low-batch, edge LLM inference, failing to deliver performance gains in large-batch, cloud-based LLM serving. We uncover a critical issue: existing INT4 quantization methods suffer from significant runtime overhead (20-90%) when dequantizing either weights or partial sums on GPUs. To address this challenge, we introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache. QoQ stands for quattuor-octo-quattuor, which represents 4-8-4 in Latin. QoQ is implemented by the QServe inference library that achieves measured speedup. The key insight driving QServe is that the efficiency of LLM serving on GPUs is critically influenced by operations on low-throughput CUDA cores. Building upon this insight, in QoQ algorithm, we introduce progressive quantization that can allow low dequantization overhead in W4A8 GEMM. Additionally, we develop SmoothAttention to effectively mitigate the accuracy degradation incurred by 4-bit KV quantization. In the QServe system, we perform compute-aware weight reordering and take advantage of register-level parallelism to reduce dequantization latency. We also make fused attention memory-bound, harnessing the performance gain brought by KV4 quantization. As a result, QServe improves the maximum achievable serving throughput of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen1.5-72B by 2.4x on A100, 3.5x on L40S, compared to TensorRT-LLM. Remarkably, QServe on L40S GPU can achieve even higher throughput than TensorRT-LLM on A100. Thus, QServe effectively reduces the dollar cost of LLM serving by 3x. Code is available at https://github.com/mit-han-lab/omniserve.

Key facts. Cited by 200 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 43 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

31 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 29

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Mistral 7Bcited by 3,065
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2024Mixtral of Expertscited by 1,785
17 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  5. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  6. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  8. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  9. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  10. 2023Microscaling Data Formats for Deep Learningcited by 132
  11. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  12. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 97
  13. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 69
  14. 2022DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generationcited by 46
  15. 2022An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformerscited by 28
  16. 2022Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Productioncited by 24
  17. 2023Dual Grained Quantization: Efficient Fine-Grained Quantization for LLMcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 87; 2026: 69.

200 papers.

Sort:

By area:

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  2. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 161
  3. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  4. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  5. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  6. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  7. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  8. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  9. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 113
  10. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  11. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  12. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
38 more
  1. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 85
  2. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 53
  3. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  4. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  5. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  6. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  7. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  8. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  9. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantizationcited by 34
  10. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  11. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  12. 20241.58-bit FLUXcited by 31
  13. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 27
  14. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  15. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 26
  16. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 25
  17. 2024Context Parallelism for Scalable Million-Token Inferencecited by 25
  18. 2025Embodied AI: From LLMs to World Modelscited by 23
  19. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Modelscited by 22
  20. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 22
  21. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 21
  22. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  23. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  24. 2025Recipes for Pre-training LLMs with MXFP8cited by 18
  25. 2025Block Rotation is All You Need for MXFP4 Quantizationcited by 18
  26. 2024MixLLM: LLM Quantization with Global Mixed-precision between Output-features and Highly-efficient System Designcited by 17
  27. 2024LiteVAR: Compressing Visual Autoregressive Modelling with Efficient Attention and Quantizationcited by 16
  28. 2025SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Modelscited by 15
  29. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  30. 2024QQQ: Quality Quattuor-Bit Quantization for Large Language Modelscited by 14
  31. 2025Accurate INT8 Training Through Dynamic Block-Level Fallbackcited by 14
  32. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 14
  33. 2026ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMscited by 14
  34. 2024LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matricescited by 11
  35. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationcited by 11
  36. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  37. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  38. 2025MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Modelscited by 11

50 most cited of 200 All 200 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Speed Odyssey for Deployable Quantization of LLMs
  2. 2023Efficient LLM Inference on CPUs
  3. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
  4. 2025CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs
  5. 2026GPU-Accelerated INT8 Quantization for KV Cache Compression in Large Language Models
  6. 2026QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
  7. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs
  8. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference