# Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm

arXiv: https://arxiv.org/abs/1808.00278 · HTML: https://mlsanity.com/paper/1808.00278/

## Key facts

- Cited by 235 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2018.

## Built on by (most cited first, 30 of 235)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation](https://mlsanity.com/paper/2111.14826/) (2021)
- [Advancing Spiking Neural Networks towards Deep Residual Learning](https://mlsanity.com/paper/2112.08954/) (2021)
- [BiBERT: Accurate Fully Binarized BERT](https://mlsanity.com/paper/2203.06390/) (2022)
- [Oscillation-free Quantization for Low-bit Vision Transformers](https://mlsanity.com/paper/2302.02210/) (2023)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [ReCU: Reviving the Dead Weights in Binary Neural Networks](https://mlsanity.com/paper/2103.12369/) (2021)
- [BiT: Robustly Binarized Multi-distilled Transformer](https://mlsanity.com/paper/2205.13016/) (2022)
- [Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2104.00676/) (2021)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization](https://mlsanity.com/paper/2502.02631/) (2025)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [OMPQ: Orthogonal Mixed Precision Quantization](https://mlsanity.com/paper/2109.07865/) (2021)
- [Learning Frequency Domain Approximation for Binary Neural Networks](https://mlsanity.com/paper/2103.00841/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [NIPQ: Noise proxy-based Integrated Pseudo-Quantization](https://mlsanity.com/paper/2206.00820/) (2022)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [Learned Smartphone ISP on Mobile NPUs with Deep Learning, Mobile AI 2021 Challenge: Report](https://mlsanity.com/paper/2105.07809/) (2021)
- [Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report](https://mlsanity.com/paper/2105.07825/) (2021)
- [PTQ4SAM: Post-Training Quantization for Segment Anything](https://mlsanity.com/paper/2405.03144/) (2024)
- [STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs](https://mlsanity.com/paper/2408.01803/) (2024)
- [Distribution Adaptive INT8 Quantization for Training CNNs](https://mlsanity.com/paper/2102.04782/) (2021)
- [BiViT: Extremely Compressed Binary Vision Transformer](https://mlsanity.com/paper/2211.07091/) (2022)
- [Faster Than Lies: Real-time Deepfake Detection using Binary Neural Networks](https://mlsanity.com/paper/2406.04932/) (2024)
- [Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training](https://mlsanity.com/paper/2206.06501/) (2022)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)

## Abstract (arXiv)

In this work, we study the 1-bit convolutional neural networks (CNNs), of which both the weights and activations are binary. While being efficient, the classification accuracy of the current 1-bit CNNs is much worse compared to their counterpart real-valued CNN models on the large-scale dataset, like ImageNet. To minimize the performance gap between the 1-bit and real-valued CNN models, we propose a novel model, dubbed Bi-Real net, which connects the real activations (after the 1-bit convolution and/or BatchNorm layer, before the sign function) to activations of the consecutive block, through an identity shortcut. Consequently, compared to the standard 1-bit CNN, the representational capability of the Bi-Real net is significantly enhanced and the additional cost on computation is negligible. Moreover, we develop a specific training algorithm including three technical novelties for 1- bit CNNs. Firstly, we derive a tight approximation to the derivative of the non-differentiable sign function with respect to activation. Secondly, we propose a magnitude-aware gradient with respect to the weight for updating the weight parameters. Thirdly, we pre-train the real-valued CNN model with a clip function, rather than the ReLU function, to better initialize the Bi-Real net. Experiments on ImageNet show that the Bi-Real net with the proposed training algorithm achieves 56.4% and 62.2% top-1 accuracy with 18 layers and 34 layers, respectively. Compared to the state-of-the-arts (e.g., XNOR Net), Bi-Real net achieves up to 10% higher top-1 accuracy with more memory saving and lower computational cost. Keywords: binary neural network, 1-bit CNNs, 1-layer-per-block