Training Quantised Neural Networks with STE Variants: the Additive Noise Annealing Algorithm
Not on the map. Why?
Abstract (arXiv)
Training quantised neural networks (QNNs) is a non-differentiable optimisation problem since weights and features are output by piecewise constant functions. The standard solution is to apply the straight-through estimator (STE), using different functions during the inference and gradient computation steps. Several STE variants have been proposed in the literature aiming to maximise the task accuracy of the trained network. In this paper, we analyse STE variants and study their impact on QNN training. We first observe that most such variants can be modelled as stochastic regularisations of stair functions; although this intuitive interpretation is not new, our rigorous discussion generalises to further variants. Then, we analyse QNNs mixing different regularisations, finding that some suitably synchronised smoothing of each layer map is required to guarantee pointwise compositional convergence to the target discontinuous function. Based on these theoretical insights, we propose additive noise annealing (ANA), a new algorithm to train QNNs encompassing standard STE and its variants as special cases. When testing ANA on the CIFAR-10 image classification benchmark, we find that the major impact on task accuracy is not due to the qualitative shape of the regularisations but to the proper synchronisation of the different STE variants used in a network, in accordance with the theoretical results.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 21 Mar 2022.
Builds on
23 of 34 references.
References · 23
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,020
- 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
- 2019Learned Step Size Quantizationcited by 628
- 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
- 2017Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networkscited by 406
11 more
- 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
- 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
- 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 152
- 2017Towards Accurate Binary Convolutional Neural Networkcited by 149
- 2017Hello Edge: Keyword Spotting on Microcontrollerscited by 110
- 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
- 2020Bayesian Bits: Unifying Quantization and Pruningcited by 94
- 2019Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networkscited by 90
- 2017Training Quantized Nets: A Deeper Understandingcited by 75
- 2018Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentationcited by 37
- 2018Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communicationcited by 14
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.