# Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution

arXiv: https://arxiv.org/abs/2012.00938 · HTML: https://mlsanity.com/paper/2012.00938/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2020.

## Built on by (most cited first, 15 of 15)

- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [Network Binarization via Contrastive Learning](https://mlsanity.com/paper/2207.02970/) (2022)
- [Optical Transformers](https://mlsanity.com/paper/2302.10360/) (2023)
- [INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold](https://mlsanity.com/paper/2204.07439/) (2022)
- [Efficient Multitask Dense Predictor via Binarization](https://mlsanity.com/paper/2405.14136/) (2024)
- [A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Network](https://mlsanity.com/paper/2403.03739/) (2024)
- [Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers](https://mlsanity.com/paper/2402.17710/) (2024)
- [BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons](https://mlsanity.com/paper/2212.14158/) (2022)
- [Binary Neural Network for Speaker Verification](https://mlsanity.com/paper/2104.02306/) (2021)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [LAB: Learnable Activation Binarizer for Binary Neural Networks](https://mlsanity.com/paper/2210.13858/) (2022)
- [Understanding weight-magnitude hyperparameters in training binary networks](https://mlsanity.com/paper/2303.02452/) (2023)
- [CycleBNN: Cyclic Precision Training in Binary Neural Networks](https://mlsanity.com/paper/2410.00050/) (2024)
- [Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks](https://mlsanity.com/paper/2601.22660/) (2026)
- [Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks](https://mlsanity.com/paper/2606.27759/) (2026)

## Abstract (arXiv)

Binarization of neural network models is considered as one of the promising methods to deploy deep neural network models on resource-constrained environments such as mobile devices. However, Binary Neural Networks (BNNs) tend to suffer from severe accuracy degradation compared to the full-precision counterpart model. Several techniques were proposed to improve the accuracy of BNNs. One of the approaches is to balance the distribution of binary activations so that the amount of information in the binary activations becomes maximum. Based on extensive analysis, in stark contrast to previous work, we argue that unbalanced activation distribution can actually improve the accuracy of BNNs. We also show that adjusting the threshold values of binary activation functions results in the unbalanced distribution of the binary activation, which increases the accuracy of BNN models. Experimental results show that the accuracy of previous BNN models (e.g. XNOR-Net and Bi-Real-Net) can be improved by simply shifting the threshold values of binary activation functions without requiring any other modification.