# ReCU: Reviving the Dead Weights in Binary Neural Networks

arXiv: https://arxiv.org/abs/2103.12369 · HTML: https://mlsanity.com/paper/2103.12369/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 58 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [Quantization Networks](https://mlsanity.com/paper/1911.09464/) (2019)
- [Towards Accurate Binary Convolutional Neural Network](https://mlsanity.com/paper/1711.11294/) (2017)
- [Deep Learning with Low Precision by Half-wave Gaussian Quantization](https://mlsanity.com/paper/1702.00953/) (2017)
- [XNOR-Net++: Improved Binary Neural Networks](https://mlsanity.com/paper/1909.13863/) (2019)
- [Towards Effective Low-bitwidth Convolutional Neural Networks](https://mlsanity.com/paper/1711.00205/) (2017)
- [Rotated Binary Neural Network](https://mlsanity.com/paper/2009.13055/) (2020)
- [Global Sparse Momentum SGD for Pruning Very Deep Neural Networks](https://mlsanity.com/paper/1909.12778/) (2019)
- [Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM](https://mlsanity.com/paper/1707.09870/) (2017)
- [Regularizing Activation Distribution for Training Binarized Deep Networks](https://mlsanity.com/paper/1904.02823/) (2019)

## Built on by (most cited first, 30 of 52)

- [PB-LLM: Partially Binarized Large Language Models](https://mlsanity.com/paper/2310.00034/) (2023)
- [BiBERT: Accurate Fully Binarized BERT](https://mlsanity.com/paper/2203.06390/) (2022)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [Faster Than Lies: Real-time Deepfake Detection using Binary Neural Networks](https://mlsanity.com/paper/2406.04932/) (2024)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [Information Theoretic Representation Distillation](https://mlsanity.com/paper/2112.00459/) (2021)
- [I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantization](https://mlsanity.com/paper/2311.10126/) (2023)
- [Lightweight Pixel Difference Networks for Efficient Visual Representation Learning](https://mlsanity.com/paper/2402.00422/) (2024)
- [Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticket](https://mlsanity.com/paper/2211.12933/) (2022)
- [Resilient Binary Neural Network](https://mlsanity.com/paper/2302.00956/) (2023)
- [Recurrent Bilinear Optimization for Binary Neural Networks](https://mlsanity.com/paper/2209.01542/) (2022)
- [TerViT: An Efficient Ternary Vision Transformer](https://mlsanity.com/paper/2201.08050/) (2022)
- [BinaryDM: Accurate Weight Binarization for Efficient Diffusion Models](https://mlsanity.com/paper/2404.05662/) (2024)
- [BiDM: Pushing the Limit of Quantization for Diffusion Models](https://mlsanity.com/paper/2412.05926/) (2024)
- [INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold](https://mlsanity.com/paper/2204.07439/) (2022)
- [BiPer: Binary Neural Networks using a Periodic Function](https://mlsanity.com/paper/2404.01278/) (2024)
- [PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models](https://mlsanity.com/paper/2502.13179/) (2025)
- [PECAN: A Product-Quantized Content Addressable Memory Network](https://mlsanity.com/paper/2208.13571/) (2022)
- [Towards Accurate Binary Spiking Neural Networks: Learning with Adaptive Gradient Modulation Mechanism](https://mlsanity.com/paper/2502.14344/) (2025)
- [BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performance](https://mlsanity.com/paper/2211.06987/) (2022)
- [GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples](https://mlsanity.com/paper/2305.07931/) (2023)
- [Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers](https://mlsanity.com/paper/2402.17710/) (2024)
- [Maximal Brain Damage Without Data or Optimization: Disrupting Neural Networks via Sign-Bit Flips](https://mlsanity.com/paper/2502.07408/) (2025)
- [BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons](https://mlsanity.com/paper/2212.14158/) (2022)
- [CTMQ: Cyclic Training of Convolutional Neural Networks with Multiple Quantization Steps](https://mlsanity.com/paper/2206.12794/) (2022)
- [E2FIF: Push the limit of Binarized Deep Imagery Super-resolution using End-to-end Full-precision Information Flow](https://mlsanity.com/paper/2207.06893/) (2022)
- [RepBNN: towards a precise Binary Neural Network with Enhanced Feature Map via Repeating](https://mlsanity.com/paper/2207.09049/) (2022)
- [Quark: An Integer RISC-V Vector Processor for Sub-Byte Quantized DNN Inference](https://mlsanity.com/paper/2302.05996/) (2023)
- [Optimizing data-flow in Binary Neural Networks](https://mlsanity.com/paper/2304.00952/) (2023)

## Abstract (arXiv)

Binary neural networks (BNNs) have received increasing attention due to their superior reductions of computation and memory. Most existing works focus on either lessening the quantization error by minimizing the gap between the full-precision weights and their binarization or designing a gradient approximation to mitigate the gradient mismatch, while leaving the "dead weights" untouched. This leads to slow convergence when training BNNs. In this paper, for the first time, we explore the influence of "dead weights" which refer to a group of weights that are barely updated during the training of BNNs, and then introduce rectified clamp unit (ReCU) to revive the "dead weights" for updating. We prove that reviving the "dead weights" by ReCU can result in a smaller quantization error. Besides, we also take into account the information entropy of the weights, and then mathematically analyze why the weight standardization can benefit BNNs. We demonstrate the inherent contradiction between minimizing the quantization error and maximizing the information entropy, and then propose an adaptive exponential scheduler to identify the range of the "dead weights". By considering the "dead weights", our method offers not only faster BNN training, but also state-of-the-art performance on CIFAR-10 and ImageNet, compared with recent methods. Code can be available at https://github.com/z-hXu/ReCU.