mlsanityData to 7 Oct 2026

Understanding weight-magnitude hyperparameters in training binary networks

arXiv 2303.02452 ↗First posted 4 Mar 2023cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Binary Neural Networks (BNNs) are compact and efficient by using binary weights instead of real-valued weights. Current BNNs use latent real-valued weights during training, where several training hyper-parameters are inherited from real-valued networks. The interpretation of several of these hyperparameters is based on the magnitude of the real-valued weights. For BNNs, however, the magnitude of binary weights is not meaningful, and thus it is unclear what these hyperparameters actually do. One example is weight-decay, which aims to keep the magnitude of real-valued weights small. Other examples are latent weight initialization, the learning rate, and learning rate decay, which influence the magnitude of the real-valued weights. The magnitude is interpretable for real-valued weights, but loses its meaning for binary weights. In this paper we offer a new interpretation of these magnitude-based hyperparameters based on higher-order gradient filtering during network optimization. Our analysis makes it possible to understand how magnitude-based hyperparameters influence the training of binary networks which allows for new optimization filters specifically designed for binary neural networks that are independent of their real-valued interpretation. Moreover, our improved understanding reduces the number of hyperparameters, which in turn eases the hyperparameter tuning effort which may lead to better hyperparameter values for improved accuracy. Code is available at https://github.com/jorisquist/Understanding-WM-HP-in-BNNs

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 4 Mar 2023.

Builds on

35 of 46 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  4. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  5. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  6. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  7. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  8. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  9. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
  10. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  11. 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
  12. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
23 more
  1. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
  2. 2019XNOR-Net++: Improved Binary Neural Networkscited by 94
  3. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  4. 2020Rotated Binary Neural Networkcited by 77
  5. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 58
  6. 2016Loss-aware Binarization of Deep Networkscited by 56
  7. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 51
  8. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 48
  9. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  10. 2020MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?cited by 32
  11. 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 22
  12. 2020Binarizing MobileNet via Evolution-based Searchingcited by 21
  13. 2018Probabilistic Binary Neural Networkscited by 20
  14. 2021Distance-aware Quantizationcited by 19
  15. 2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
  16. 2019MoBiNet: A Mobile Binary Network for Image Classificationcited by 15
  17. 2020Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networkscited by 15
  18. 2020Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distributioncited by 15
  19. 2021Equal Bits: Enforcing Equally Distributed Binary Network Weightscited by 9
  20. 2020Stochastic Gradient Variance Reduction by Solving a Filtering Problemcited by 3
  21. 2021A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networkscited by 3
  22. 2021Elastic-Link for Binarized Neural Networkcited by 3
  23. 2022How to train accurate BNNs for embedded systems?cited by 1

Built on by

Not cited yet.