mlsanityData to 14 Oct 2026

Custom Gradient Estimators are Straight-Through Estimators in Disguise

arXiv 2405.05171 ↗First posted 8 May 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Quantization-aware training comes with a fundamental challenge: the derivative of quantization functions such as rounding are zero almost everywhere and nonexistent elsewhere. Various differentiable approximations of quantization functions have been proposed to address this issue. In this paper, we prove that when the learning rate is sufficiently small, a large class of weight gradient estimators is equivalent with the straight through estimator (STE). Specifically, after swapping in the STE and adjusting both the weight initialization and the learning rate in SGD, the model will train in almost exactly the same way as it did with the original gradient estimator. Moreover, we show that for adaptive learning rate algorithms like Adam, the same result can be seen without any modifications to the weight initialization and learning rate. We experimentally show that these results hold for both a small convolutional model trained on the MNIST dataset and for a ResNet50 model trained on ImageNet.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 36 of its 63 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

36 of 63 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  7. 2016An overview of gradient descent optimization algorithmscited by 1,033
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  9. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  10. 2015Cyclical Learning Rates for Training Neural Networkscited by 629
  11. 2019Learned Step Size Quantizationcited by 628
  12. 2016Learning to learn by gradient descent by gradient descentcited by 620
24 more
  1. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  2. 2021A White Paper on Neural Network Quantizationcited by 472
  3. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  4. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  5. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  6. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  7. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 160
  8. 2019Quantization Networkscited by 153
  9. 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
  10. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  11. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  12. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  13. 2020Rotated Binary Neural Networkcited by 77
  14. 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
  15. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 60
  16. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 49
  17. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  18. 2021A comprehensive review of Binary Neural Networkcited by 45
  19. 2021Learning Frequency Domain Approximation for Binary Neural Networkscited by 38
  20. 2022Power-of-Two Quantization for Low Bitwidth and Hardware Compliant Neural Networkscited by 26
  21. 2022Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Trainingcited by 26
  22. 2019Accurate and Compact Convolutional Neural Networks with Trained Binarizationcited by 20
  23. 2021Distance-aware Quantizationcited by 19
  24. 2018Regularized Binary Network Trainingcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2025LittleBit: Ultra Low-Bit Quantization via Latent Factorizationcited by 11
  2. 2025Improving the Straight-Through Estimator with Zeroth-Order Informationcited by 5
  3. 2024Continuous Approximations for Improving Quantization Aware Training of LLMscited by 2
  4. 2025Compact Rule-Based Classifier Learning via Gradient Descentcited by 2
  5. 2025Learning Binary Sampling Patterns for Single-Pixel Imaging using Bilevel Optimisationcited by 1
  6. 2026TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Gradient-Free Training of Quantized Neural Networks
  2. 2023On Uniform Scalar Quantization for Learned Image Compression
  3. 2022Error-aware Quantization through Noise Tempering
  4. 2025Progressive Element-wise Gradient Estimation for Neural Network Quantization
  5. 2025High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator
  6. 2025Adaptive Distribution-aware Quantization for Mixed-Precision Neural Networks
  7. 2025CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
  8. 2024Robust Training of Neural Networks at Arbitrary Precision and Sparsity