# Towards Efficient Training for Neural Network Quantization

arXiv: https://arxiv.org/abs/1912.10207 · HTML: https://mlsanity.com/paper/1912.10207/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2019.

## Built on by (most cited first, 16 of 16)

- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Teachers Do More Than Teach: Compressing Image-to-Image Models](https://mlsanity.com/paper/2103.03467/) (2021)
- [Q-ViT: Fully Differentiable Quantization for Vision Transformer](https://mlsanity.com/paper/2201.07703/) (2022)
- [Bit-Mixer: Mixed-precision networks with runtime bit-width selection](https://mlsanity.com/paper/2103.17267/) (2021)
- [Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution](https://mlsanity.com/paper/2106.02295/) (2021)
- [Data Quality-aware Mixed-precision Quantization via Hybrid Reinforcement Learning](https://mlsanity.com/paper/2302.04453/) (2023)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization](https://mlsanity.com/paper/2311.06798/) (2023)
- [Single-path Bit Sharing for Automatic Loss-aware Model Compression](https://mlsanity.com/paper/2101.04935/) (2021)
- [SamurAI: A Versatile IoT Node With Event-Driven Wake-Up and Embedded ML Acceleration](https://mlsanity.com/paper/2304.13726/) (2023)
- [Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization](https://mlsanity.com/paper/2105.07331/) (2021)
- [PIM-QAT: Neural Network Quantization for Processing-In-Memory (PIM) Systems](https://mlsanity.com/paper/2209.08617/) (2022)
- [RCT: Resource Constrained Training for Edge AI](https://mlsanity.com/paper/2103.14493/) (2021)
- [AdaQAT: Adaptive Bit-Width Quantization-Aware Training](https://mlsanity.com/paper/2404.16876/) (2024)
- [Efficient Neural Compression with Inference-time Decoding](https://mlsanity.com/paper/2406.06237/) (2024)
- [Nearly Lossless Adaptive Bit Switching](https://mlsanity.com/paper/2502.01199/) (2025)

## Abstract (arXiv)

Quantization reduces computation costs of neural networks but suffers from performance degeneration. Is this accuracy drop due to the reduced capacity, or inefficient training during the quantization procedure? After looking into the gradient propagation process of neural networks by viewing the weights and intermediate activations as random variables, we discover two critical rules for efficient training. Recent quantization approaches violates the two rules and results in degenerated convergence. To deal with this problem, we propose a simple yet effective technique, named scale-adjusted training (SAT), to comply with the discovered rules and facilitates efficient training. We also analyze the quantization error introduced in calculating the gradient in the popular parameterized clipping activation (PACT) technique. Through SAT together with gradient-calibrated PACT, quantized models obtain comparable or even better performance than their full-precision counterparts, achieving state-of-the-art accuracy with consistent improvement over previous quantization methods on a wide spectrum of models including MobileNet-V1/V2 and PreResNet-50.