# Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies

arXiv: https://arxiv.org/abs/2209.01404 · HTML: https://mlsanity.com/paper/2209.01404/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 3 Sep 2022.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [End-to-End Human Pose and Mesh Reconstruction with Transformers](https://mlsanity.com/paper/2012.09760/) (2020)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions](https://mlsanity.com/paper/2003.03488/) (2020)
- [Forward and Backward Information Retention for Accurate Binary Neural Networks](https://mlsanity.com/paper/1909.10788/) (2019)
- [Towards Accurate Binary Convolutional Neural Network](https://mlsanity.com/paper/1711.11294/) (2017)
- [CycleMLP: A MLP-like Architecture for Dense Prediction](https://mlsanity.com/paper/2107.10224/) (2021)
- [Training Binary Neural Networks with Real-to-Binary Convolutions](https://mlsanity.com/paper/2003.11535/) (2020)
- [Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions](https://mlsanity.com/paper/1711.08141/) (2017)
- [Network Quantization with Element-wise Gradient Scaling](https://mlsanity.com/paper/2104.00903/) (2021)
- [S2-MLP: Spatial-Shift MLP Architecture for Vision](https://mlsanity.com/paper/2106.07477/) (2021)
- [Rotated Binary Neural Network](https://mlsanity.com/paper/2009.13055/) (2020)
- [Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation](https://mlsanity.com/paper/2103.08273/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [Searching for Low-Bit Weights in Quantized Neural Networks](https://mlsanity.com/paper/2009.08695/) (2020)
- [Learnable Companding Quantization for Accurate Low-bit Neural Networks](https://mlsanity.com/paper/2103.07156/) (2021)

## Built on by (most cited first, 6 of 6)

- [SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms](https://mlsanity.com/paper/2406.03287/) (2024)
- [SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking](https://mlsanity.com/paper/2407.04752/) (2024)
- [Lightweight Pixel Difference Networks for Efficient Visual Representation Learning](https://mlsanity.com/paper/2402.00422/) (2024)
- [GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples](https://mlsanity.com/paper/2305.07931/) (2023)
- [BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomials](https://mlsanity.com/paper/2312.08937/) (2023)
- [BOLD: Boolean Logic Deep Learning](https://mlsanity.com/paper/2405.16339/) (2024)

## Abstract (arXiv)

Existing Binary Neural Networks (BNNs) mainly operate on local convolutions with binarization function. However, such simple bit operations lack the ability of modeling contextual dependencies, which is critical for learning discriminative deep representations in vision models. In this work, we tackle this issue by presenting new designs of binary neural modules, which enables BNNs to learn effective contextual dependencies. First, we propose a binary multi-layer perceptron (MLP) block as an alternative to binary convolution blocks to directly model contextual dependencies. Both short-range and long-range feature dependencies are modeled by binary MLPs, where the former provides local inductive bias and the latter breaks limited receptive field in binary convolutions. Second, to improve the robustness of binary models with contextual dependencies, we compute the contextual dynamic embeddings to determine the binarization thresholds in general binary convolutional blocks. Armed with our binary MLP blocks and improved binary convolution, we build the BNNs with explicit Contextual Dependency modeling, termed as BCDNet. On the standard ImageNet-1K classification benchmark, the BCDNet achieves 72.3% Top-1 accuracy and outperforms leading binary methods by a large margin. In particular, the proposed BCDNet exceeds the state-of-the-art ReActNet-A by 2.9% Top-1 accuracy with similar operations. Our code is available at https://github.com/Sense-GVT/BCDN