# Bi-Real Net: Binarizing Deep Network Towards Real-Network Performance

arXiv: https://arxiv.org/abs/1811.01335 · HTML: https://mlsanity.com/paper/1811.01335/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2018.

## Built on by (most cited first, 30 of 42)

- [PB-LLM: Partially Binarized Large Language Models](https://mlsanity.com/paper/2310.00034/) (2023)
- [Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation](https://mlsanity.com/paper/2111.14826/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Towards Accurate Post-training Quantization for Diffusion Models](https://mlsanity.com/paper/2305.18723/) (2023)
- [Q-VLM: Post-training Quantization for Large Vision-Language Models](https://mlsanity.com/paper/2410.08119/) (2024)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Q-DETR: An Efficient Low-Bit Quantized Detection Transformer](https://mlsanity.com/paper/2304.00253/) (2023)
- [Carrying out CNN Channel Pruning in a White Box](https://mlsanity.com/paper/2104.11883/) (2021)
- [Lipschitz Continuity Retained Binary Neural Network](https://mlsanity.com/paper/2207.06540/) (2022)
- [Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket](https://mlsanity.com/paper/2211.12933/) (2022)
- [Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models](https://mlsanity.com/paper/2406.12311/) (2024)
- [Quantized Spike-driven Transformer](https://mlsanity.com/paper/2501.13492/) (2025)
- [Network Binarization via Contrastive Learning](https://mlsanity.com/paper/2207.02970/) (2022)
- [IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectors](https://mlsanity.com/paper/2210.03477/) (2022)
- [BiDM: Pushing the Limit of Quantization for Diffusion Models](https://mlsanity.com/paper/2412.05926/) (2024)
- [Lottery Jackpots Exist in Pre-trained Models](https://mlsanity.com/paper/2104.08700/) (2021)
- [BiPer: Binary Neural Networks using a Periodic Function](https://mlsanity.com/paper/2404.01278/) (2024)
- [Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks](https://mlsanity.com/paper/2110.09195/) (2021)
- [Binary and Ternary Natural Language Generation](https://mlsanity.com/paper/2306.01841/) (2023)
- [Efficient Multitask Dense Predictor via Binarization](https://mlsanity.com/paper/2405.14136/) (2024)
- [XnODR and XnIDR: Two Accurate and Fast Fully Connected Layers For Convolutional Neural Networks](https://mlsanity.com/paper/2111.10854/) (2021)
- [Self-Distribution Binary Neural Networks](https://mlsanity.com/paper/2103.02394/) (2021)
- [BiFeat: Supercharge GNN Training via Graph Feature Quantization](https://mlsanity.com/paper/2207.14696/) (2022)
- [Vertical Layering of Quantized Neural Networks for Heterogeneous Inference](https://mlsanity.com/paper/2212.05326/) (2022)
- [A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks](https://mlsanity.com/paper/2104.05124/) (2021)
- [Compacting Binary Neural Networks by Sparse Kernel Selection](https://mlsanity.com/paper/2303.14470/) (2023)
- [Binarizing by Classification: Is soft function really necessary?](https://mlsanity.com/paper/2205.07433/) (2022)
- [Filter Pruning based on Information Capacity and Independence](https://mlsanity.com/paper/2303.03645/) (2023)

## Abstract (arXiv)

In this paper, we study 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary. While efficient, the lacking of representational capability and the training difficulty impede 1-bit CNNs from performing as well as real-valued networks. We propose Bi-Real net with a novel training algorithm to tackle these two challenges. To enhance the representational capability, we propagate the real-valued activations generated by each 1-bit convolution via a parameter-free shortcut. To address the training difficulty, we propose a training algorithm using a tighter approximation to the derivative of the sign function, a magnitude-aware gradient for weight updating, a better initialization method, and a two-step scheme for training a deep network. Experiments on ImageNet show that an 18-layer Bi-Real net with the proposed training algorithm achieves 56.4% top-1 classification accuracy, which is 10% higher than the state-of-the-arts (e.g., XNOR-Net) with greater memory saving and lower computational cost. Bi-Real net is also the first to scale up 1-bit CNNs to an ultra-deep network with 152 layers, and achieves 64.5% top-1 accuracy on ImageNet. A 50-layer Bi-Real net shows comparable performance to a real-valued network on the depth estimation task with only a 0.3% accuracy gap.