# Back to Simplicity: How to Train Accurate BNNs from Scratch?

arXiv: https://arxiv.org/abs/1906.08637 · HTML: https://mlsanity.com/paper/1906.08637/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jun 2019.

## Built on by (most cited first, 24 of 24)

- [ReCU: Reviving the Dead Weights in Binary Neural Networks](https://mlsanity.com/paper/2103.12369/) (2021)
- [Learning Frequency Domain Approximation for Binary Neural Networks](https://mlsanity.com/paper/2103.00841/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [Faster Than Lies: Real-time Deepfake Detection using Binary Neural Networks](https://mlsanity.com/paper/2406.04932/) (2024)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [SiMaN: Sign-to-Magnitude Network Binarization](https://mlsanity.com/paper/2102.07981/) (2021)
- [Enabling Binary Neural Network Training on the Edge](https://mlsanity.com/paper/2102.04270/) (2021)
- [ZippyPoint: Fast Interest Point Detection, Description, and Matching through Mixed Precision Discretization](https://mlsanity.com/paper/2203.03610/) (2022)
- [Bias-Variance Tradeoffs in Single-Sample Binary Gradient Estimators](https://mlsanity.com/paper/2110.03549/) (2021)
- [Enabling On-device Continual Learning with Binary Neural Networks](https://mlsanity.com/paper/2401.09916/) (2024)
- [A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks](https://mlsanity.com/paper/2104.05124/) (2021)
- [Elastic-Link for Binarized Neural Network](https://mlsanity.com/paper/2112.10149/) (2021)
- [On-Device Learning with Binary Neural Networks](https://mlsanity.com/paper/2308.15308/) (2023)
- [Optimizing data-flow in Binary Neural Networks](https://mlsanity.com/paper/2304.00952/) (2023)
- [Effective Neural Network L0 Regularization With BinMask](https://mlsanity.com/paper/2304.11237/) (2023)
- [Gradient-Based Interpretability Methods and Binarized Neural Networks](https://mlsanity.com/paper/2106.12569/) (2021)
- [Binary Early-Exit Network for Adaptive Inference on Low-Resource Devices](https://mlsanity.com/paper/2206.09029/) (2022)
- [Binary domain generalization for sparsifying binary neural networks](https://mlsanity.com/paper/2306.13515/) (2023)
- [Bit-wise Training of Neural Network Weights](https://mlsanity.com/paper/2202.09571/) (2022)
- [Sparsifying Binary Networks](https://mlsanity.com/paper/2207.04974/) (2022)
- [Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices](https://mlsanity.com/paper/2207.05784/) (2022)
- [LAB: Learnable Activation Binarizer for Binary Neural Networks](https://mlsanity.com/paper/2210.13858/) (2022)
- [What to Do When Your Discrete Optimization Is the Size of a Neural Network?](https://mlsanity.com/paper/2402.10339/) (2024)
- [CycleBNN: Cyclic Precision Training in Binary Neural Networks](https://mlsanity.com/paper/2410.00050/) (2024)

## Abstract (arXiv)

Binary Neural Networks (BNNs) show promising progress in reducing computational and memory costs but suffer from substantial accuracy degradation compared to their real-valued counterparts on large-scale datasets, e.g., ImageNet. Previous work mainly focused on reducing quantization errors of weights and activations, whereby a series of approximation methods and sophisticated training tricks have been proposed. In this work, we make several observations that challenge conventional wisdom. We revisit some commonly used techniques, such as scaling factors and custom gradients, and show that these methods are not crucial in training well-performing BNNs. On the contrary, we suggest several design principles for BNNs based on the insights learned and demonstrate that highly accurate BNNs can be trained from scratch with a simple training strategy. We propose a new BNN architecture BinaryDenseNet, which significantly surpasses all existing 1-bit CNNs on ImageNet without tricks. In our experiments, BinaryDenseNet achieves 18.6% and 7.6% relative improvement over the well-known XNOR-Network and the current state-of-the-art Bi-Real Net in terms of top-1 accuracy on ImageNet, respectively.