Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers
Not on the map. Why?
Abstract (arXiv)
In neural network binarization, BinaryConnect (BC) and its variants are considered the standard. These methods apply the sign function in their forward pass and their respective gradients are backpropagated to update the weights. However, the derivative of the sign function is zero whenever defined, which consequently freezes training. Therefore, implementations of BC (e.g., BNN) usually replace the derivative of sign in the backward computation with identity or other approximate gradient alternatives. Although such practice works well empirically, it is largely a heuristic or ''training trick.'' We aim at shedding some light on these training tricks from the optimization perspective. Building from existing theory on ProxConnect (PC, a generalization of BC), we (1) equip PC with different forward-backward quantizers and obtain ProxConnect++ (PC++) that includes existing binarization techniques as special cases; (2) derive a principled way to synthesize forward-backward quantizers with automatic theoretical guarantees; (3) illustrate our theory by proposing an enhanced binarization algorithm BNN++; (4) conduct image classification experiments on CNNs and vision transformers, and empirically verify that BNN++ generally achieves competitive results on binarizing these models.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 27 Feb 2024.
Builds on
46 of 58 references.
References · 46
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
- 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
- 2019Learned Step Size Quantizationcited by 604
- 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
34 more
- 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
- 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
- 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
- 2015Deep Learning with Limited Numerical Precisioncited by 368
- 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
- 2021Post-Training Quantization for Vision Transformercited by 274
- 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
- 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
- 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
- 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 180
- 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
- 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
- 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
- 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
- 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
- 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
- 2019Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networkscited by 87
- 2020Rotated Binary Neural Networkcited by 77
- 2022BiBERT: Accurate Fully Binarized BERTcited by 70
- 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
- 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 58
- 2018ProxQuant: Quantized Neural Networks via Proximal Operatorscited by 56
- 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 51
- 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
- 2022AdaBin: Improving Binary Neural Networks with Adaptive Binary Setscited by 38
- 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
- 2021Leveraging redundancy in attention with Reuse Transformerscited by 33
- 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
- 2020Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distributioncited by 15
- 2018Regularized Binary Network Trainingcited by 13
- 2021Multi-Dimensional Model Compression of Vision Transformercited by 13
- 2021Demystifying and Generalizing BinaryConnectcited by 11
- 2022Is Integer Arithmetic Enough for Deep Learning Training?cited by 11
- 2022TerViT: An Efficient Ternary Vision Transformercited by 10
Built on by
Citing papers per year: 2025: 4.
4 papers.