mlsanityData to 14 Oct 2026

Bi-ViT: Pushing the Limit of Vision Transformer Quantization

arXiv 2305.12354 ↗First posted 21 May 2023cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Vision transformers (ViTs) quantization offers a promising prospect to facilitate deploying large pre-trained networks on resource-limited devices. Fully-binarized ViTs (Bi-ViT) that pushes the quantization of ViTs to its limit remain largely unexplored and a very challenging task yet, due to their unacceptable performance. Through extensive empirical analyses, we identify the severe drop in ViT binarization is caused by attention distortion in self-attention, which technically stems from the gradient vanishing and ranking disorder. To address these issues, we first introduce a learnable scaling factor to reactivate the vanished gradients and illustrate its effectiveness through theoretical and experimental analyses. We then propose a ranking-aware distillation method to rectify the disordered ranking in a teacher-student framework. Bi-ViT achieves significant improvements over popular DeiT and Swin backbones in terms of Top-1 accuracy and FLOPs. For example, with DeiT-Tiny and Swin-Tiny, our method significantly outperforms baselines by 22.1% and 21.4% respectively, while 61.5x and 56.1x theoretical acceleration in terms of FLOPs compared with real-valued counterparts on ImageNet.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

37 of 45 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2017Mask R-CNNcited by 7,277
  9. 2017Squeeze-and-Excitation Networkscited by 4,789
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
25 more
  1. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,276
  2. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  3. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  4. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  5. 2019Learned Step Size Quantizationcited by 628
  6. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  7. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  8. 2022DeiT III: Revenge of the ViTcited by 419
  9. 2021Post-Training Quantization for Vision Transformercited by 278
  10. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  11. 2013Predicting Parameters in Deep Learningcited by 224
  12. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 184
  13. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 177
  14. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 112
  15. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  16. 2020Rotated Binary Neural Networkcited by 77
  17. 2022BiBERT: Accurate Fully Binarized BERTcited by 73
  18. 2022CF-ViT: A General Coarse-to-Fine Method for Vision Transformercited by 63
  19. 2021Learning Efficient Vision Transformers via Fine-Grained Manifold Distillationcited by 60
  20. 2016Local Binary Convolutional Neural Networkscited by 38
  21. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 37
  22. 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 24
  23. 2022Recurrent Bilinear Optimization for Binary Neural Networkscited by 12
  24. 2022IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectorscited by 12
  25. 2021Fine-grained Data Distribution Alignment for Post-Training Quantizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025BHViT: Binarized Hybrid Vision Transformercited by 5
  2. 2024An Analysis on Quantizing Diffusion Transformerscited by 3
  3. 2024Mixed Non-linear Quantization for Vision Transformerscited by 3
  4. 2026Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networkscited by 3
  5. 2024Error Diffusion: Post Training Quantization with Block-Scaled Number Formats for Neural Networkscited by 1
  6. 2025SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exitcited by 1
  7. 2025TeTRA-VPR: A Ternary Transformer Approach for Compact Visual Place Recognitioncited by 1
  8. 2025Precision Neural Network Quantization via Learnable Adaptive Modulescited by 1
  9. 2025High-Fidelity Differential-information Driven Binary Vision Transformercited by 0
  10. 2025Assessing the Potential for Catastrophic Failure in Dynamic Post-Training Quantizationcited by 0
  11. 2026FTerViT: Fully Ternary Vision Transformercited by 0
  12. 2026ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformerscited by 0
1 more
  1. 2026Learning 1-Bit LiDAR-based Localization with Auxiliary Objectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023BinaryViT: Towards Efficient and Accurate Binary Vision Transformers
  2. 2024ViT-1.58b: Mobile Vision Transformers in the 1-bit Era
  3. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
  4. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
  5. 2024Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer
  6. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantization
  7. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers
  8. 2024ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers