# A White Paper on Neural Network Quantization

arXiv: https://arxiv.org/abs/2106.08295 · HTML: https://mlsanity.com/paper/2106.08295/

## Key facts

- Cited by 472 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 15 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [LSQ+: Improving low-bit quantization through learnable offsets and better initialization](https://mlsanity.com/paper/2004.09576/) (2020)
- [Bayesian Bits: Unifying Quantization and Pruning](https://mlsanity.com/paper/2005.07093/) (2020)
- [And the Bit Goes Down: Revisiting the Quantization of Neural Networks](https://mlsanity.com/paper/1907.05686/) (2019)
- [Same, Same But Different - Recovering Neural Network Quantization Error Through Weight Factorization](https://mlsanity.com/paper/1902.01917/) (2019)
- [Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces](https://mlsanity.com/paper/2012.08859/) (2020)
- [Fighting Quantization Bias With Bias](https://mlsanity.com/paper/1906.03193/) (2019)
- [A Quantization-Friendly Separable Convolution for MobileNets](https://mlsanity.com/paper/1803.08607/) (2018)
- [One Weight Bitwidth to Rule Them All](https://mlsanity.com/paper/2008.09916/) (2020)
- [DSConv: Efficient Convolution Operator](https://mlsanity.com/paper/1901.01928/) (2019)

## Built on by (most cited first, 30 of 472)

- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Q-Diffusion: Quantizing Diffusion Models](https://mlsanity.com/paper/2302.04304/) (2023)
- [Post-training Quantization on Diffusion Models](https://mlsanity.com/paper/2211.15736/) (2022)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [Understanding and Overcoming the Challenges of Efficient Transformer Quantization](https://mlsanity.com/paper/2109.12948/) (2021)
- [Overcoming Oscillations in Quantization-Aware Training](https://mlsanity.com/paper/2203.11086/) (2022)
- [ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation](https://mlsanity.com/paper/2406.02540/) (2024)
- [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://mlsanity.com/paper/2304.07493/) (2023)
- [Loki: Low-rank Keys for Efficient Sparse Attention](https://mlsanity.com/paper/2406.02542/) (2024)
- [FP8 Quantization: The Power of the Exponent](https://mlsanity.com/paper/2208.09225/) (2022)
- [When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions](https://mlsanity.com/paper/2306.15546/) (2023)
- [GPTVQ: The Blessing of Dimensionality for LLM Quantization](https://mlsanity.com/paper/2402.15319/) (2024)
- [PTQ4DiT: Post-training Quantization for Diffusion Transformers](https://mlsanity.com/paper/2405.16005/) (2024)
- [Evaluating Quantized Large Language Models](https://mlsanity.com/paper/2402.18158/) (2024)
- [PD-Quant: Post-Training Quantization based on Prediction Difference Metric](https://mlsanity.com/paper/2212.07048/) (2022)
- [Stateful Large Language Model Serving with Pensieve](https://mlsanity.com/paper/2312.05516/) (2023)
- [Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs](https://mlsanity.com/paper/2410.13835/) (2024)
- [Implicit Neural Video Compression](https://mlsanity.com/paper/2112.11312/) (2021)
- [ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization](https://mlsanity.com/paper/2208.14286/) (2022)
- [Intriguing Properties of Quantization at Scale](https://mlsanity.com/paper/2305.19268/) (2023)
- [FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization](https://mlsanity.com/paper/2306.00317/) (2023)
- [Pruning vs Quantization: Which is Better?](https://mlsanity.com/paper/2307.02973/) (2023)
- [ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization](https://mlsanity.com/paper/2502.02631/) (2025)
- [Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators](https://mlsanity.com/paper/2302.08469/) (2023)
- [MixDQ: Memory-Efficient Few-Step Text-to-Image Diffusion Models with Metric-Decoupled Mixed Precision Quantization](https://mlsanity.com/paper/2405.17873/) (2024)
- [Low-Rank Quantization-Aware Training for LLMs](https://mlsanity.com/paper/2406.06385/) (2024)
- [QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models](https://mlsanity.com/paper/2310.16795/) (2023)

## Abstract (arXiv)

While neural networks have advanced the frontiers in many applications, they often come at a high computational cost. Reducing the power and latency of neural network inference is key if we want to integrate modern networks into edge devices with strict power and compute requirements. Neural network quantization is one of the most effective ways of achieving these savings but the additional noise it induces can lead to accuracy degradation. In this white paper, we introduce state-of-the-art algorithms for mitigating the impact of quantization noise on the network's performance while maintaining low-bit weights and activations. We start with a hardware motivated introduction to quantization and then consider two main classes of algorithms: Post-Training Quantization (PTQ) and Quantization-Aware-Training (QAT). PTQ requires no re-training or labelled data and is thus a lightweight push-button approach to quantization. In most cases, PTQ is sufficient for achieving 8-bit quantization with close to floating-point accuracy. QAT requires fine-tuning and access to labeled training data but enables lower bit quantization with competitive results. For both solutions, we provide tested pipelines based on existing literature and extensive experimentation that lead to state-of-the-art performance for common deep learning models and tasks.