# Forward and Backward Information Retention for Accurate Binary Neural Networks

arXiv: https://arxiv.org/abs/1909.10788 · HTML: https://mlsanity.com/paper/1909.10788/

## Key facts

- Cited by 169 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2019.

## Built on by (most cited first, 30 of 169)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization](https://mlsanity.com/paper/2203.05740/) (2022)
- [BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://mlsanity.com/paper/2402.04291/) (2024)
- [Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World](https://mlsanity.com/paper/2103.01050/) (2021)
- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [Network Quantization with Element-wise Gradient Scaling](https://mlsanity.com/paper/2104.00903/) (2021)
- [PB-LLM: Partially Binarized Large Language Models](https://mlsanity.com/paper/2310.00034/) (2023)
- [BiBERT: Accurate Fully Binarized BERT](https://mlsanity.com/paper/2203.06390/) (2022)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Diversifying Sample Generation for Accurate Data-Free Quantization](https://mlsanity.com/paper/2103.01049/) (2021)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [ReCU: Reviving the Dead Weights in Binary Neural Networks](https://mlsanity.com/paper/2103.12369/) (2021)
- [BiT: Robustly Binarized Multi-distilled Transformer](https://mlsanity.com/paper/2205.13016/) (2022)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [A comprehensive review of Binary Neural Network](https://mlsanity.com/paper/2110.06804/) (2021)
- [Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy](https://mlsanity.com/paper/2307.16867/) (2023)
- [IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization](https://mlsanity.com/paper/2111.09136/) (2021)
- [Learning Frequency Domain Approximation for Binary Neural Networks](https://mlsanity.com/paper/2103.00841/) (2021)
- [AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets](https://mlsanity.com/paper/2208.08084/) (2022)
- [Convolutional Differentiable Logic Gate Networks](https://mlsanity.com/paper/2411.04732/) (2024)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusion](https://mlsanity.com/paper/2404.08406/) (2024)
- [Towards Real-world X-ray Security Inspection: A High-Quality Benchmark and Lateral Inhibition Module for Prohibited Items Detection](https://mlsanity.com/paper/2108.09917/) (2021)
- [BiViT: Extremely Compressed Binary Vision Transformer](https://mlsanity.com/paper/2211.07091/) (2022)
- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [Faster Than Lies: Real-time Deepfake Detection using Binary Neural Networks](https://mlsanity.com/paper/2406.04932/) (2024)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)

## Abstract (arXiv)

Weight and activation binarization is an effective approach to deep neural network compression and can accelerate the inference by leveraging bitwise operations. Although many binarization methods have improved the accuracy of the model by minimizing the quantization error in forward propagation, there remains a noticeable performance gap between the binarized model and the full-precision one. Our empirical study indicates that the quantization brings information loss in both forward and backward propagation, which is the bottleneck of training accurate binary neural networks. To address these issues, we propose an Information Retention Network (IR-Net) to retain the information that consists in the forward activations and backward gradients. IR-Net mainly relies on two technical contributions: (1) Libra Parameter Binarization (Libra-PB): simultaneously minimizing both quantization error and information loss of parameters by balanced and standardized weights in forward propagation; (2) Error Decay Estimator (EDE): minimizing the information loss of gradients by gradually approximating the sign function in backward propagation, jointly considering the updating ability and accurate gradients. We are the first to investigate both forward and backward processes of binary networks from the unified information perspective, which provides new insight into the mechanism of network binarization. Comprehensive experiments with various network structures on CIFAR-10 and ImageNet datasets manifest that the proposed IR-Net can consistently outperform state-of-the-art quantization methods.