mlsanityData to 14 Oct 2026

Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training

arXiv 2308.06689 ↗First posted 13 Aug 2023cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Binarization of neural networks is a dominant paradigm in neural networks compression. The pioneering work BinaryConnect uses Straight Through Estimator (STE) to mimic the gradients of the sign function, but it also causes the crucial inconsistency problem. Most of the previous methods design different estimators instead of STE to mitigate it. However, they ignore the fact that when reducing the estimating error, the gradient stability will decrease concomitantly. These highly divergent gradients will harm the model training and increase the risk of gradient vanishing and gradient exploding. To fully take the gradient stability into consideration, we present a new perspective to the BNNs training, regarding it as the equilibrium between the estimating error and the gradient stability. In this view, we firstly design two indicators to quantitatively demonstrate the equilibrium phenomenon. In addition, in order to balance the estimating error and the gradient stability well, we revise the original straight through estimator and propose a power function based estimator, Rectified Straight Through Estimator (ReSTE for short). Comparing to other estimators, ReSTE is rational and capable of flexibly balancing the estimating error with the gradient stability. Extensive experiments on CIFAR-10 and ImageNet datasets show that ReSTE has excellent performance and surpasses the state-of-the-art methods without any auxiliary modules or losses.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 50 of its 60 references that are in the map. First posted on arXiv 13 Aug 2023.

Builds on

50 of 60 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  7. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
38 more
  1. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  2. 2019Relational Knowledge Distillationcited by 889
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  4. 2019Contrastive Representation Distillationcited by 748
  5. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  6. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  7. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  8. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  9. 2022Decoupled Knowledge Distillationcited by 388
  10. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  11. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  12. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  13. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  14. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  15. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  16. 2017Towards Accurate Binary Convolutional Neural Networkcited by 149
  17. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 117
  18. 2019XNOR-Net++: Improved Binary Neural Networkscited by 97
  19. 2020ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgettingcited by 85
  20. 2021Vision Transformer Pruningcited by 79
  21. 2020Rotated Binary Neural Networkcited by 77
  22. 2017Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMMcited by 61
  23. 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
  24. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 60
  25. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 59
  26. 2016Loss-aware Binarization of Deep Networkscited by 56
  27. 2017Introduction to Tensor Decompositions and their Applications in Machine Learningcited by 46
  28. 2021Learning Frequency Domain Approximation for Binary Neural Networkscited by 38
  29. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  30. 2016Ternary Neural Networks for Resource-Efficient AI Applicationscited by 33
  31. 2018Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagationcited by 29
  32. 2018From Hashing to CNNs: Training BinaryWeight Networks via Hashingcited by 23
  33. 2017Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantizationcited by 21
  34. 2019Mirror Descent View for Neural Network Quantizationcited by 16
  35. 2019Bayesian Optimized 1-Bit CNNscited by 14
  36. 2022Lipschitz Continuity Retained Binary Neural Networkcited by 14
  37. 2022Recurrent Bilinear Optimization for Binary Neural Networkscited by 12
  38. 2015Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 10; 2026: 1.

19 papers.

Sort:

By area:

  1. 2025Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Taskscited by 28
  2. 2024Binarized Low-light Raw Video Enhancementcited by 17
  3. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  4. 2024BiDM: Pushing the Limit of Quantization for Diffusion Modelscited by 11
  5. 2025Panorama Generation From NFoV Image Done Rightcited by 11
  6. 2024BiPer: Binary Neural Networks using a Periodic Functioncited by 9
  7. 2023On the Opportunities of Green Computing: A Surveycited by 7
  8. 2025BiMaCoSR: Binary One-Step Diffusion Model Leveraging Flexible Matrix Compression for Real Super-Resolutioncited by 5
  9. 2025CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Trainingcited by 4
  10. 20241-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bitcited by 3
  11. 2024OvSW: Overcoming Silent Weights for Accurate Binary Neural Networkscited by 2
  12. 2025GDNSQ: Gradual Differentiable Noise Scale Quantization for Low-bit Neural Networkscited by 2
7 more
  1. 2024Fast and Slow Gradient Approximation for Binary Neural Network Optimizationcited by 0
  2. 2025Hyperbolic Binary Neural Networkcited by 0
  3. 2025Expanding-and-Shrinking Binary Neural Networkscited by 0
  4. 2025NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devicescited by 0
  5. 2025End-to-end fully-binarized network design: from Generic Learned Thermometer to Block Pruningcited by 0
  6. 2025AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuningcited by 0
  7. 2026Training single-electron and single-photon stochastic physical neural networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021AdaSTE: An Adaptive Straight-Through Estimator to Train Binary Neural Networks
  2. 2026Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
  3. 2021A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
  4. 2026SURGE: Surrogate Gradient Adaptation in Binary Neural Networks
  5. 2024Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers
  6. 2023Injecting Logical Constraints into Neural Networks via Straight-Through Estimators
  7. 2022Binarizing by Classification: Is soft function really necessary?
  8. 2025Beyond Discreteness: Sample Complexity Analysis of Straight-Through Estimator for 1-bit Quantization