# Training Binary Neural Networks with Real-to-Binary Convolutions

arXiv: https://arxiv.org/abs/2003.11535 · HTML: https://mlsanity.com/paper/2003.11535/

## Key facts

- Cited by 112 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2020.

## Built on by (most cited first, 30 of 112)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [PB-LLM: Partially Binarized Large Language Models](https://mlsanity.com/paper/2310.00034/) (2023)
- [Advancing Spiking Neural Networks towards Deep Residual Learning](https://mlsanity.com/paper/2112.08954/) (2021)
- [BiBERT: Accurate Fully Binarized BERT](https://mlsanity.com/paper/2203.06390/) (2022)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [BiT: Robustly Binarized Multi-distilled Transformer](https://mlsanity.com/paper/2205.13016/) (2022)
- [Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2104.00676/) (2021)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization](https://mlsanity.com/paper/2111.09136/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [Quantization Backdoors to Deep Learning Commercial Frameworks](https://mlsanity.com/paper/2108.09187/) (2021)
- [BiViT: Extremely Compressed Binary Vision Transformer](https://mlsanity.com/paper/2211.07091/) (2022)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks](https://mlsanity.com/paper/2109.12872/) (2021)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [Bit-Mixer: Mixed-precision networks with runtime bit-width selection](https://mlsanity.com/paper/2103.17267/) (2021)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [EcoFormer: Energy-Saving Attention with Linear Complexity](https://mlsanity.com/paper/2209.09004/) (2022)
- [MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models](https://mlsanity.com/paper/2412.11549/) (2024)
- [SiMaN: Sign-to-Magnitude Network Binarization](https://mlsanity.com/paper/2102.07981/) (2021)
- ["BNN - BN = ?": Training Binary Neural Networks without Batch Normalization](https://mlsanity.com/paper/2104.08215/) (2021)
- [Lightweight Pixel Difference Networks for Efficient Visual Representation Learning](https://mlsanity.com/paper/2402.00422/) (2024)
- [Generative Zero-shot Network Quantization](https://mlsanity.com/paper/2101.08430/) (2021)
- [S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration](https://mlsanity.com/paper/2102.08946/) (2021)
- [Hard Sample Matters a Lot in Zero-Shot Quantization](https://mlsanity.com/paper/2303.13826/) (2023)
- [Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks](https://mlsanity.com/paper/2104.07085/) (2021)
- [Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket](https://mlsanity.com/paper/2211.12933/) (2022)
- [Dynamic Binary Neural Network by learning channel-wise thresholds](https://mlsanity.com/paper/2110.05185/) (2021)

## Abstract (arXiv)

This paper shows how to train binary networks to within a few percent points (~ 3-5 %) of the full precision counterpart. We first show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances and carefully adjusting the optimization procedure. Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution, additional significant accuracy gains can be obtained. We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in a data-driven manner, by using the real-valued activations, available during inference prior to the binarization process, for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet and reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 accuracy on CIFAR-100 and ImageNet respectively when using a ResNet-18 architecture. Code available at https://github.com/brais-martinez/real2binary.