mlsanityData to 7 Oct 2026

Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview

arXiv 2409.11650 ↗First posted 18 Sep 2024cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper provides a comprehensive overview of the principles, challenges, and methodologies associated with quantizing large-scale neural network models. As neural networks have evolved towards larger and more complex architectures to address increasingly sophisticated tasks, the computational and energy costs have escalated significantly. We explore the necessity and impact of model size growth, highlighting the performance benefits as well as the computational challenges and environmental considerations. The core focus is on model quantization as a fundamental approach to mitigate these challenges by reducing model size and improving efficiency without substantially compromising accuracy. We delve into various quantization techniques, including both post-training quantization (PTQ) and quantization-aware training (QAT), and analyze several state-of-the-art algorithms such as LLM-QAT, PEQA(L4Q), ZeroQuant, SmoothQuant, and others. Through comparative analysis, we examine how these methods address issues like outliers, importance weighting, and activation quantization, ultimately contributing to more sustainable and accessible deployment of large-scale models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 53 of its 57 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

53 of 57 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  9. 2019Learned Step Size Quantizationcited by 604
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  12. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
38 more
  1. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  6. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  10. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  11. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  13. 2021Post-Training Quantization for Vision Transformercited by 274
  14. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  15. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  16. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  17. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  18. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  19. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  20. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  21. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  22. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  23. 2018Training and Inference with Integers in Deep Neural Networkscited by 84
  24. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  25. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 83
  26. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  27. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
  28. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  29. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  30. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  31. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 58
  32. 2024Low-Rank Quantization-Aware Training for LLMscited by 42
  33. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 41
  34. 2018Post-training 4-bit quantization of convolution networks for rapid-deploymentcited by 38
  35. 2019Training Quantized Neural Networks with a Full-precision Auxiliary Modulecited by 35
  36. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 20
  37. 2024EasyQuant: An Efficient Data-free Quantization Algorithm for LLMscited by 17
  38. 2021Post-Training Sparsity-Aware Quantizationcited by 16

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  2. 2025AI Governance through Marketscited by 5