# Quantizing deep convolutional networks for efficient inference: A whitepaper

arXiv: https://arxiv.org/abs/1806.08342 · HTML: https://mlsanity.com/paper/1806.08342/

## Key facts

- Cited by 521 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2018.

## Built on by (most cited first, 30 of 521)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [COIN: COmpression with Implicit Neural representations](https://mlsanity.com/paper/2103.03123/) (2021)
- [Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models](https://mlsanity.com/paper/2209.13325/) (2022)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [Understanding and Overcoming the Challenges of Efficient Transformer Quantization](https://mlsanity.com/paper/2109.12948/) (2021)
- [RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers](https://mlsanity.com/paper/2212.08254/) (2022)
- [CompGS: Smaller and Faster Gaussian Splatting with Vector Quantization](https://mlsanity.com/paper/2311.18159/) (2023)
- [Overcoming Oscillations in Quantization-Aware Training](https://mlsanity.com/paper/2203.11086/) (2022)
- [Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation](https://mlsanity.com/paper/2307.15337/) (2023)
- [HNeRV: A Hybrid Neural Representation for Videos](https://mlsanity.com/paper/2304.02633/) (2023)
- [I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2207.01405/) (2022)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [FP8 Quantization: The Power of the Exponent](https://mlsanity.com/paper/2208.09225/) (2022)
- [COIN++: Neural Compression Across Modalities](https://mlsanity.com/paper/2201.12904/) (2022)
- [EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models](https://mlsanity.com/paper/2310.03270/) (2023)
- [Evaluating Quantized Large Language Models](https://mlsanity.com/paper/2402.18158/) (2024)
- [PD-Quant: Post-Training Quantization based on Prediction Difference Metric](https://mlsanity.com/paper/2212.07048/) (2022)
- [Compressing Volumetric Radiance Fields to 1 MB](https://mlsanity.com/paper/2211.16386/) (2022)
- [VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference](https://mlsanity.com/paper/2102.04503/) (2021)
- [Patch Similarity Aware Data-Free Quantization for Vision Transformers](https://mlsanity.com/paper/2203.02250/) (2022)
- [Implicit Neural Video Compression](https://mlsanity.com/paper/2112.11312/) (2021)
- [Online Continual Learning for Embedded Devices](https://mlsanity.com/paper/2203.10681/) (2022)

## Abstract (arXiv)

We present an overview of techniques for quantizing convolutional neural networks for inference with integer weights and activations. Per-channel quantization of weights and per-layer quantization of activations to 8-bits of precision post-training produces classification accuracies within 2% of floating point networks for a wide variety of CNN architectures. Model sizes can be reduced by a factor of 4 by quantizing weights to 8-bits, even when 8-bit arithmetic is not supported. This can be achieved with simple, post training quantization of weights.We benchmark latencies of quantized networks on CPUs and DSPs and observe a speedup of 2x-3x for quantized implementations compared to floating point on CPUs. Speedups of up to 10x are observed on specialized processors with fixed point SIMD capabilities, like the Qualcomm QDSPs with HVX. Quantization-aware training can provide further improvements, reducing the gap to floating point to 1% at 8-bit precision. Quantization-aware training also allows for reducing the precision of weights to four bits with accuracy losses ranging from 2% to 10%, with higher accuracy drop for smaller networks.We introduce tools in TensorFlow and TensorFlowLite for quantizing convolutional networks and review best practices for quantization-aware training to obtain high accuracy with quantized weights and activations. We recommend that per-channel quantization of weights and per-layer quantization of activations be the preferred quantization scheme for hardware acceleration and kernel optimization. We also propose that future processors and hardware accelerators for optimized inference support precisions of 4, 8 and 16 bits.