mlsanityData to 14 Oct 2026

Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks

arXiv 2601.22660 ↗First posted 30 Jan 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We investigate progressive freezing as an alternative to straight-through estimators (STE) for training binary networks from scratch. Under controlled training conditions, we find that while global progressive freezing works for binary-weight networks, it fails for full binary neural networks due to activation-induced gradient blockades. We introduce StoMPP (Stochastic Masked Partial Progressive Binarization), which uses layerwise stochastic masking to progressively replace differentiable clipped weights/activations with hard binary step functions, while only backpropagating through the unfrozen (clipped) subset (i.e., no straight-through estimator). Under a matched minimal training recipe, StoMPP improves accuracy over a BinaryConnect-style STE baseline, with gains that increase with depth (e.g., for ResNet-50 BNN: +18.0 on CIFAR-10, +13.5 on CIFAR-100, and +3.8 on ImageNet; for ResNet-18: +3.1, +4.7, and +1.3). For binary-weight networks, StoMPP achieves 91.2% accuracy on CIFAR-10 and 69.5% on CIFAR-100 with ResNet-50. We analyze training dynamics under progressive freezing, revealing non-monotonic convergence and improved depth scaling under binarization constraints.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

38 of 47 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  5. 2019Learned Step Size Quantizationcited by 628
  6. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  7. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  8. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  9. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  10. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  11. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  12. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
26 more
  1. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  2. 2017Towards Accurate Binary Convolutional Neural Networkcited by 149
  3. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  4. 2019XNOR-Net++: Improved Binary Neural Networkscited by 97
  5. 2019Defensive Quantization: When Efficiency Meets Robustnesscited by 95
  6. 2019Regularizing Activation Distribution for Training Binarized Deep Networkscited by 61
  7. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 60
  8. 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 59
  9. 2016Loss-aware Binarization of Deep Networkscited by 56
  10. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 52
  11. 2021Soft then Hard: Rethinking the Quantization in Neural Image Compressioncited by 51
  12. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  13. 2019QKD: Quantization-aware Knowledge Distillationcited by 44
  14. 2018ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multiplierscited by 42
  15. 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
  16. 2018Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagationcited by 29
  17. 2019Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagationcited by 28
  18. 2018BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weightscited by 24
  19. 2020BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activationscited by 23
  20. 2017Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantizationcited by 21
  21. 2019Learning low-precision neural networks without Straight-Through Estimator(STE)cited by 16
  22. 2020Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distributioncited by 15
  23. 2019Self-Binarizing Networkscited by 11
  24. 2019Structured Binary Neural Networks for Image Recognitioncited by 10
  25. 2024BiPer: Binary Neural Networks using a Periodic Functioncited by 9
  26. 2024OvSW: Overcoming Silent Weights for Accurate Binary Neural Networkscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks
  2. 2026Pruning Binarized Neural Networks: A Dedicated Framework and Globally Weighted Algorithms
  3. 2021Enabling Binary Neural Network Training on the Edge
  4. 2026Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training
  5. 2024Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers
  6. 2023Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training
  7. 2026Variational Filtering for Bayesian Last Layers with an Independent Binary Probit Surrogate
  8. 2022Partial Binarization of Neural Networks for Budget-Aware Efficient Learning