# Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations

arXiv: https://arxiv.org/abs/1908.04680 · HTML: https://mlsanity.com/paper/1908.04680/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2019.

## Built on by (most cited first, 18 of 18)

- [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://mlsanity.com/paper/2304.07493/) (2023)
- [SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation](https://mlsanity.com/paper/2202.07471/) (2022)
- [ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization](https://mlsanity.com/paper/2208.14286/) (2022)
- [Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs](https://mlsanity.com/paper/2309.05516/) (2023)
- [Towards Efficient Post-training Quantization of Pre-trained Language Models](https://mlsanity.com/paper/2109.15082/) (2021)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization](https://mlsanity.com/paper/2109.00212/) (2021)
- [Class-based Quantization for Neural Networks](https://mlsanity.com/paper/2211.14928/) (2022)
- [TEQ: Trainable Equivalent Transformation for Quantization of LLMs](https://mlsanity.com/paper/2310.10944/) (2023)
- [Towards Efficient Task-Driven Model Reprogramming with Foundation Models](https://mlsanity.com/paper/2304.02263/) (2023)
- [Vertical Layering of Quantized Neural Networks for Heterogeneous Inference](https://mlsanity.com/paper/2212.05326/) (2022)
- [Compact Model Training by Low-Rank Projection with Energy Transfer](https://mlsanity.com/paper/2204.05566/) (2022)
- [Efficient Adaptive Activation Rounding for Post-Training Quantization](https://mlsanity.com/paper/2208.11945/) (2022)
- [RedBit: An End-to-End Flexible Framework for Evaluating the Accuracy of Quantized CNNs](https://mlsanity.com/paper/2301.06193/) (2023)
- [Dual Precision Quantization for Efficient and Accurate Deep Neural Networks Inference](https://mlsanity.com/paper/2505.14638/) (2025)
- [RepQ: Generalizing Quantization-Aware Training for Re-Parametrized Architectures](https://mlsanity.com/paper/2311.05317/) (2023)
- [PQCAD-DM: Progressive Quantization and Calibration-Assisted Distillation for Extremely Efficient Diffusion Model](https://mlsanity.com/paper/2506.16776/) (2025)
- [Progressive Multi-Ancestor Bit-Depth Distillation](https://mlsanity.com/paper/2610.04100/) (2026)

## Abstract (arXiv)

This paper tackles the problem of training a deep convolutional neural network of both low-bitwidth weights and activations. Optimizing a low-precision network is very challenging due to the non-differentiability of the quantizer, which may result in substantial accuracy loss. To address this, we propose three practical approaches, including (i) progressive quantization; (ii) stochastic precision; and (iii) joint knowledge distillation to improve the network training. First, for progressive quantization, we propose two schemes to progressively find good local minima. Specifically, we propose to first optimize a net with quantized weights and subsequently quantize activations. This is in contrast to the traditional methods which optimize them simultaneously. Furthermore, we propose a second progressive quantization scheme which gradually decreases the bit-width from high-precision to low-precision during training. Second, to alleviate the excessive training burden due to the multi-round training stages, we further propose a one-stage stochastic precision strategy to randomly sample and quantize sub-networks while keeping other parts in full-precision. Finally, we adopt a novel learning scheme to jointly train a full-precision model alongside the low-precision one. By doing so, the full-precision model provides hints to guide the low-precision model training and significantly improves the performance of the low-precision network. Extensive experiments on various datasets (e.g., CIFAR-100, ImageNet) show the effectiveness of the proposed methods.