# Self-Binarizing Networks

arXiv: https://arxiv.org/abs/1902.00730 · HTML: https://mlsanity.com/paper/1902.00730/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2019.

## Built on by (most cited first, 11 of 11)

- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks](https://mlsanity.com/paper/2104.05124/) (2021)
- [BCNN: Binary Complex Neural Network](https://mlsanity.com/paper/2104.10044/) (2021)
- [Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations](https://mlsanity.com/paper/2107.01400/) (2021)
- [GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm](https://mlsanity.com/paper/2206.03291/) (2022)
- [RedBit: An End-to-End Flexible Framework for Evaluating the Accuracy of Quantized CNNs](https://mlsanity.com/paper/2301.06193/) (2023)
- [Searching Similarity Measure for Binarized Neural Networks](https://mlsanity.com/paper/2206.03325/) (2022)
- [Interpretable Classification via a Rule Network with Selective Logical Operators](https://mlsanity.com/paper/2408.11918/) (2024)
- [Layerwise Progressive Freezing Enables STE-Free Training of Deep Binary Neural Networks](https://mlsanity.com/paper/2601.22660/) (2026)
- [A Composite Activation Function for Learning Stable Binary Representations](https://mlsanity.com/paper/2605.11558/) (2026)
- [Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks](https://mlsanity.com/paper/2606.27759/) (2026)

## Abstract (arXiv)

We present a method to train self-binarizing neural networks, that is, networks that evolve their weights and activations during training to become binary. To obtain similar binary networks, existing methods rely on the sign activation function. This function, however, has no gradients for non-zero values, which makes standard backpropagation impossible. To circumvent the difficulty of training a network relying on the sign activation function, these methods alternate between floating-point and binary representations of the network during training, which is sub-optimal and inefficient. We approach the binarization task by training on a unique representation involving a smooth activation function, which is iteratively sharpened during training until it becomes a binary representation equivalent to the sign activation function. Additionally, we introduce a new technique to perform binary batch normalization that simplifies the conventional batch normalization by transforming it into a simple comparison operation. This is unlike existing methods, which are forced to the retain the conventional floating-point-based batch normalization. Our binary networks, apart from displaying advantages of lower memory and computation as compared to conventional floating-point and binary networks, also show higher classification accuracy than existing state-of-the-art methods on multiple benchmark datasets.