mlsanityData to 7 Oct 2026

GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples

arXiv 2305.07931 ↗First posted 13 May 2023cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Vision Transformer (ViT) has performed remarkably in various computer vision tasks. Nonetheless, affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limited number of training samples. In addition, ViT generally demands heavy computing resources, which limit its deployment on resource-constrained devices. As a type of model-compression method, model binarization is potentially a good choice to solve the above problems. Compared with the full-precision one, the model with the binarization method replaces complex tensor multiplication with simple bit-wise binary operations and represents full-precision model parameters and activations with only 1-bit ones, which potentially solves the problem of model size and computational complexity, respectively. In this paper, we investigate a binarized ViT model. Empirically, we observe that the existing binarization technology designed for Convolutional Neural Networks (CNN) cannot migrate well to a ViT's binarization task. We also find that the decline of the accuracy of the binary ViT model is mainly due to the information loss of the Attention module and the Value vector. Therefore, we propose a novel model binarization technique, called Group Superposition Binarization (GSB), to deal with these issues. Furthermore, in order to further improve the performance of the binarization model, we have investigated the gradient calculation procedure in the binarization process and derived more proper gradient calculation equations for GSB to reduce the influence of gradient mismatch. Then, the knowledge distillation technique is introduced to alleviate the performance degradation caused by model binarization. Analytically, model binarization can limit the parameters search space during parameter updates while training a model....

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 13 May 2023.

Builds on

41 of 53 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  7. 2021Transformer in Transformercited by 632
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  9. 2020Overfitting in adversarially robust deep learningcited by 504
  10. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  11. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
  12. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
29 more
  1. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  2. 2017Towards Accurate Binary Convolutional Neural Networkcited by 148
  3. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  4. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 115
  5. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
  6. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  7. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  8. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78
  9. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
  10. 2020Rotated Binary Neural Networkcited by 77
  11. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  12. 2022Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networkscited by 55
  13. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 55
  14. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 51
  15. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
  16. 2021Hard Sample Aware Noise Robust Learning for Histopathology Image Classificationcited by 48
  17. 2021Learning Frequency Domain Approximation for Binary Neural Networkscited by 38
  18. 2018Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentationcited by 37
  19. 2022Training-free Transformer Architecture Searchcited by 33
  20. 2022Locality Guidance for Improving Vision Transformers on Tiny Datasetscited by 33
  21. 2022Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasetscited by 28
  22. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  23. 2022Training Vision Transformers with Only 2040 Imagescited by 22
  24. 2023BiBench: Benchmarking and Analyzing Network Binarizationcited by 22
  25. 2021"BNN - BN = ?": Training Binary Neural Networks without Batch Normalizationcited by 17
  26. 2022Accumulated Trivial Attention Matters in Vision Transformers on Small Datasetscited by 15
  27. 2022Recurrent Bilinear Optimization for Binary Neural Networkscited by 11
  28. 2016Overcoming Challenges in Fixed Point Training of Deep Convolutional Networkscited by 9
  29. 2022Towards Accurate Binary Neural Networks via Modeling Contextual Dependenciescited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 2.

4 papers.

  1. 2023RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentationcited by 24
  2. 2025BHViT: Binarized Hybrid Vision Transformercited by 5
  3. 2024Sparsity- and Hybridity-Inspired Visual Parameter-Efficient Fine-Tuning for Medical Diagnosiscited by 3
  4. 2025High-Fidelity Differential-information Driven Binary Vision Transformercited by 0