# Equal Bits: Enforcing Equally Distributed Binary Network Weights

arXiv: https://arxiv.org/abs/2112.03406 · HTML: https://mlsanity.com/paper/2112.03406/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 40 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [What's Hidden in a Randomly Weighted Neural Network?](https://mlsanity.com/paper/1911.13299/) (2019)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://mlsanity.com/paper/1905.01067/) (2019)
- [Proving the Lottery Ticket Hypothesis: Pruning is All You Need](https://mlsanity.com/paper/2002.00585/) (2020)
- [Binary Neural Networks: A Survey](https://mlsanity.com/paper/2004.03333/) (2020)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [Towards Accurate Binary Convolutional Neural Network](https://mlsanity.com/paper/1711.11294/) (2017)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [XNOR-Net++: Improved Binary Neural Networks](https://mlsanity.com/paper/1909.13863/) (2019)
- [Rotated Binary Neural Network](https://mlsanity.com/paper/2009.13055/) (2020)
- [Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection](https://mlsanity.com/paper/2003.01794/) (2020)
- [Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization](https://mlsanity.com/paper/1906.02107/) (2019)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [Searching for Low-Bit Weights in Quantized Neural Networks](https://mlsanity.com/paper/2009.08695/) (2020)
- [Training Binary Neural Networks through Learning with Noisy Supervision](https://mlsanity.com/paper/2010.04871/) (2020)

## Built on by (most cited first, 9 of 9)

- [Lightweight Pixel Difference Networks for Efficient Visual Representation Learning](https://mlsanity.com/paper/2402.00422/) (2024)
- [A&B BNN: Add&Bit-Operation-Only Hardware-Friendly Binary Neural Network](https://mlsanity.com/paper/2403.03739/) (2024)
- [BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons](https://mlsanity.com/paper/2212.14158/) (2022)
- [End-to-End Sensitivity-Based Filter Pruning](https://mlsanity.com/paper/2204.07412/) (2022)
- [How to train accurate BNNs for embedded systems?](https://mlsanity.com/paper/2206.12322/) (2022)
- [Training Multi-Layer Binary Neural Networks With Local Binary Error Signals](https://mlsanity.com/paper/2412.00119/) (2024)
- [Pruning by Active Attention Manipulation](https://mlsanity.com/paper/2210.11114/) (2022)
- [Understanding weight-magnitude hyperparameters in training binary networks](https://mlsanity.com/paper/2303.02452/) (2023)
- [Neural Network Compression using Binarization and Few Full-Precision Weights](https://mlsanity.com/paper/2306.08960/) (2023)

## Abstract (arXiv)

Binary networks are extremely efficient as they use only two symbols to define the network: +1,-1. One can make the prior distribution of these symbols a design choice. The recent IR-Net of Qin et al. argues that imposing a Bernoulli distribution with equal priors (equal bit ratios) over the binary weights leads to maximum entropy and thus minimizes information loss. However, prior work cannot precisely control the binary weight distribution during training, and therefore cannot guarantee maximum entropy. Here, we show that quantizing using optimal transport can guarantee any bit ratio, including equal ratios. We investigate experimentally that equal bit ratios are indeed preferable and show that our method leads to optimization benefits. We show that our quantization method is effective when compared to state-of-the-art binarization methods, even when using binary weight pruning.