# Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks

arXiv: https://arxiv.org/abs/1808.05240 · HTML: https://mlsanity.com/paper/1808.05240/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2018.

## Built on by (most cited first, 21 of 21)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization](https://mlsanity.com/paper/2406.00800/) (2024)
- [Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss](https://mlsanity.com/paper/2109.02100/) (2021)
- [COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization](https://mlsanity.com/paper/2403.07134/) (2024)
- [Globally Optimal Training of Neural Networks with Threshold Activation Functions](https://mlsanity.com/paper/2303.03382/) (2023)
- [Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization](https://mlsanity.com/paper/2103.02904/) (2021)
- [Feature Affinity Assisted Knowledge Distillation and Quantization of Deep Neural Networks on Label-Free Data](https://mlsanity.com/paper/2302.10899/) (2023)
- [CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization](https://mlsanity.com/paper/2501.18475/) (2025)
- [Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/2109.03513/) (2021)
- [Automatic Sparse Connectivity Learning for Neural Networks](https://mlsanity.com/paper/2201.05020/) (2022)
- [High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator](https://mlsanity.com/paper/2510.10693/) (2025)
- [Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations](https://mlsanity.com/paper/2109.00095/) (2021)
- [GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks](https://mlsanity.com/paper/2102.09298/) (2021)
- [Beyond Discreteness: Sample Complexity Analysis of Straight-Through Estimator for 1-bit Quantization](https://mlsanity.com/paper/2505.18113/) (2025)
- [Differentiable Architecture Pruning for Transfer Learning](https://mlsanity.com/paper/2107.03375/) (2021)
- [RedBit: An End-to-End Flexible Framework for Evaluating the Accuracy of Quantized CNNs](https://mlsanity.com/paper/2301.06193/) (2023)
- [Boosting Binary Neural Networks via Dynamic Thresholds Learning](https://mlsanity.com/paper/2211.02292/) (2022)
- [AutoQNN: An End-to-End Framework for Automatically Quantizing Neural Networks](https://mlsanity.com/paper/2304.03782/) (2023)
- [Integrating Pruning with Quantization for Efficient Deep Neural Networks Compression](https://mlsanity.com/paper/2509.04244/) (2025)

## Abstract (arXiv)

Quantized deep neural networks (QDNNs) are attractive due to their much lower memory storage and faster inference speed than their regular full precision counterparts. To maintain the same performance level especially at low bit-widths, QDNNs must be retrained. Their training involves piecewise constant activation functions and discrete weights, hence mathematical challenges arise. We introduce the notion of coarse gradient and propose the blended coarse gradient descent (BCGD) algorithm, for training fully quantized neural networks. Coarse gradient is generally not a gradient of any function but an artificial ascent direction. The weight update of BCGD goes by coarse gradient correction of a weighted average of the full precision weights and their quantization (the so-called blending), which yields sufficient descent in the objective value and thus accelerates the training. Our experiments demonstrate that this simple blending technique is very effective for quantization at extremely low bit-width such as binarization. In full quantization of ResNet-18 for ImageNet classification task, BCGD gives 64.36% top-1 accuracy with binary weights across all layers and 4-bit adaptive activation. If the weights in the first and last layers are kept in full precision, this number increases to 65.46%. As theoretical justification, we show convergence analysis of coarse gradient descent for a two-linear-layer neural network model with Gaussian input data, and prove that the expected coarse gradient correlates positively with the underlying true gradient.