mlsanityData to 7 Oct 2026

Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey

arXiv 2405.00314 ↗First posted 1 May 2024cs.LG cs.AI cs.ARcited by 13

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) have recently garnered considerable attention, emerging as a promising alternative to convolutional neural networks (CNNs) in several vision-related applications. However, their large model sizes and high computational and memory demands hinder deployment, especially on resource-constrained devices. This underscores the necessity of algorithm-hardware co-design specific to ViTs, aiming to optimize their performance by tailoring both the algorithmic structure and the underlying hardware accelerator to each other's strengths. Model quantization, by converting high-precision numbers to lower-precision, reduces the computational demands and memory needs of ViTs, allowing the creation of hardware specifically optimized for these quantized algorithms, boosting efficiency. This article provides a comprehensive survey of ViTs quantization and its hardware acceleration. We first delve into the unique architectural attributes of ViTs and their runtime characteristics. Subsequently, we examine the fundamental principles of model quantization, followed by a comparative analysis of the state-of-the-art quantization techniques for ViTs. Additionally, we explore the hardware acceleration of quantized ViTs, highlighting the importance of hardware-friendly algorithm design. In conclusion, this article will discuss ongoing challenges and future research paths. We consistently maintain the related open-source materials at https://github.com/DD-DuDa/awesome-vit-quantization-acceleration.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 68 of its 83 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

68 of 83 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  12. 2021Transformers in Vision: A Surveycited by 1,009
38 more
  1. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  2. 2020Pre-Trained Image Processing Transformercited by 891
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  4. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  5. 2020A Survey on Visual Transformercited by 653
  6. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  7. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
  8. 2021Post-Training Quantization for Vision Transformercited by 274
  9. 2023A Survey on Model Compression for Large Language Modelscited by 263
  10. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  11. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  12. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  13. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  14. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  15. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  16. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  17. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  18. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  19. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  20. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  21. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  22. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  23. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  24. 2021Vision Transformer Pruningcited by 77
  25. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  26. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  27. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  28. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  29. 2020EasyQuant: Post-training Quantization via Scale Optimizationcited by 57
  30. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 55
  31. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 55
  32. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 52
  33. 2024A Survey on Transformer Compressioncited by 44
  34. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  35. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  36. 2020Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Aheadcited by 34
  37. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 28
  38. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  2. 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
  3. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  4. 2025CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimizationcited by 7
  5. 2024Qua2SeDiMo: Quantifiable Quantization Sensitivity of Diffusion Modelscited by 6
  6. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospectscited by 4
  7. 2024Mixed Non-linear Quantization for Vision Transformerscited by 3
  8. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  9. 2024Computer Vision Model Compression Techniques for Embedded Systems: A Surveycited by 1
  10. 2025I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentationcited by 1
  11. 2025DistrAttention: An Efficient and Flexible Self-Attention Mechanism on Modern GPUscited by 0
  12. 2026MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformerscited by 0
1 more
  1. 2026Low-Rank Ternary Adaptation for Fine-Tuning Transformerscited by 0