# ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions

arXiv: https://arxiv.org/abs/2003.03488 · HTML: https://mlsanity.com/paper/2003.03488/

## Key facts

- Cited by 184 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2020.

## Built on by (most cited first, 30 of 184)

- [ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation](https://mlsanity.com/paper/2406.02540/) (2024)
- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [PB-LLM: Partially Binarized Large Language Models](https://mlsanity.com/paper/2310.00034/) (2023)
- [Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation](https://mlsanity.com/paper/2111.14826/) (2021)
- [Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space](https://mlsanity.com/paper/2201.00814/) (2022)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Training Transformers with 4-bit Integers](https://mlsanity.com/paper/2306.11987/) (2023)
- [ReCU: Reviving the Dead Weights in Binary Neural Networks](https://mlsanity.com/paper/2103.12369/) (2021)
- [BiT: Robustly Binarized Multi-distilled Transformer](https://mlsanity.com/paper/2205.13016/) (2022)
- [Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study](https://mlsanity.com/paper/2104.00676/) (2021)
- [A Fast Knowledge Distillation Framework for Visual Recognition](https://mlsanity.com/paper/2112.01528/) (2021)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy](https://mlsanity.com/paper/2307.16867/) (2023)
- [Learning Frequency Domain Approximation for Binary Neural Networks](https://mlsanity.com/paper/2103.00841/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [Quantization Backdoors to Deep Learning Commercial Frameworks](https://mlsanity.com/paper/2108.09187/) (2021)
- [BiViT: Extremely Compressed Binary Vision Transformer](https://mlsanity.com/paper/2211.07091/) (2022)
- [VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer](https://mlsanity.com/paper/2201.06618/) (2022)
- [Faster Than Lies: Real-time Deepfake Detection using Binary Neural Networks](https://mlsanity.com/paper/2406.04932/) (2024)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [Quantization without Tears](https://mlsanity.com/paper/2411.13918/) (2024)
- [Q-DETR: An Efficient Low-Bit Quantized Detection Transformer](https://mlsanity.com/paper/2304.00253/) (2023)
- [Binarized Spectral Compressive Imaging](https://mlsanity.com/paper/2305.10299/) (2023)
- [Basic Binary Convolution Unit for Binarized Image Restoration Network](https://mlsanity.com/paper/2210.00405/) (2022)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks](https://mlsanity.com/paper/2203.03844/) (2022)
- [MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models](https://mlsanity.com/paper/2412.11549/) (2024)

## Abstract (arXiv)

In this paper, we propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost. We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts, bypassing all the intermediate convolutional layers including the downsampling layers. This baseline network strikes a good trade-off between accuracy and efficiency, achieving superior performance than most of existing binary networks at approximately half of the computational cost. Through extensive experiments and analysis, we observed that the performance of binary networks is sensitive to activation distribution variations. Based on this important observation, we propose to generalize the traditional Sign and PReLU functions, denoted as RSign and RPReLU for the respective generalized functions, to enable explicit learning of the distribution reshape and shift at near-zero extra cost. Lastly, we adopt a distributional loss to further enforce the binary network to learn similar output distributions as those of a real-valued network. We show that after incorporating all these ideas, the proposed ReActNet outperforms all the state-of-the-arts by a large margin. Specifically, it outperforms Real-to-Binary Net and MeliusNet29 by 4.0% and 3.6% respectively for the top-1 accuracy and also reduces the gap to its real-valued counterpart to within 3.0% top-1 accuracy on ImageNet dataset. Code and models are available at: https://github.com/liuzechun/ReActNet.