# In-Hindsight Quantization Range Estimation for Quantized Training

arXiv: https://arxiv.org/abs/2105.04246 · HTML: https://mlsanity.com/paper/2105.04246/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 10 May 2021.

## Builds on (most cited first, 17 of 17)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Data-Free Quantization Through Weight Equalization and Bias Correction](https://mlsanity.com/paper/1906.04721/) (2019)
- [Scalable Methods for 8-bit Training of Neural Networks](https://mlsanity.com/paper/1805.11046/) (2018)
- [Training and Inference with Integers in Deep Neural Networks](https://mlsanity.com/paper/1802.04680/) (2018)
- [Towards Unified INT8 Training for Convolutional Neural Network](https://mlsanity.com/paper/1912.12607/) (2019)
- [A Statistical Framework for Low-bitwidth Training of Deep Neural Networks](https://mlsanity.com/paper/2010.14298/) (2020)
- [Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers](https://mlsanity.com/paper/1909.02384/) (2019)
- [Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm](https://mlsanity.com/paper/1812.11732/) (2018)

## Built on by (most cited first, 7 of 7)

- [FP8 versus INT8 for efficient deep learning inference](https://mlsanity.com/paper/2303.17951/) (2023)
- [Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats](https://mlsanity.com/paper/2112.10769/) (2021)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Hadamard Domain Training with Integers for Class Incremental Quantized Learning](https://mlsanity.com/paper/2310.03675/) (2023)
- [AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks](https://mlsanity.com/paper/2211.03741/) (2022)
- [Toward INT4 Fixed-Point Training via Exploring Quantization Error for Gradients](https://mlsanity.com/paper/2407.12637/) (2024)
- [MicroQonv: Reshaping Convolution Tensors for Efficient Microscaling in Training and Inference](https://mlsanity.com/paper/2609.28358/) (2026)

## Abstract (arXiv)

Quantization techniques applied to the inference of deep neural networks have enabled fast and efficient execution on resource-constraint devices. The success of quantization during inference has motivated the academic community to explore fully quantized training, i.e. quantizing back-propagation as well. However, effective gradient quantization is still an open problem. Gradients are unbounded and their distribution changes significantly during training, which leads to the need for dynamic quantization. As we show, dynamic quantization can lead to significant memory overhead and additional data traffic slowing down training. We propose a simple alternative to dynamic quantization, in-hindsight range estimation, that uses the quantization ranges estimated on previous iterations to quantize the present. Our approach enables fast static quantization of gradients and activations while requiring only minimal hardware support from the neural network accelerator to keep track of output statistics in an online fashion. It is intended as a drop-in replacement for estimating quantization ranges and can be used in conjunction with other advances in quantized training. We compare our method to existing methods for range estimation from the quantized training literature and demonstrate its effectiveness with a range of architectures, including MobileNetV2, on image classification benchmarks (Tiny ImageNet & ImageNet).