mlsanityData to 7 Oct 2026

Post-Training Quantization for Vision Transformer

arXiv 2106.14156 ↗First posted 27 Jun 2021cs.CVcited by 274

Not on the map. Why?

Abstract (arXiv)

Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights and inputs, respectively. To preserve the functionality of the attention mechanism, we introduce a ranking loss into the conventional quantization objective that aims to keep the relative order of the self-attention results after quantization. Moreover, we thoroughly analyze the relationship between quantization loss of different layers and the feature diversity, and explore a mixed-precision quantization scheme by exploiting the nuclear norm of each attention map and output feature. The effectiveness of the proposed method is verified on several benchmark models and datasets, which outperforms the state-of-the-art post-training quantization algorithms. For instance, we can obtain an 81.29% top-1 accuracy using DeiT-B model on ImageNet dataset with about 8-bit quantization.

Key facts. Cited by 274 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 27 Jun 2021.

Builds on

32 of 39 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020End-to-End Object Detection with Transformerscited by 7,384
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2020Pre-Trained Image Processing Transformercited by 891
  8. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  9. 2021Transformer in Transformercited by 632
  10. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  11. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  12. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
20 more
  1. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  2. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  3. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  4. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  5. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  6. 2016Trained Ternary Quantizationcited by 203
  7. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  8. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  9. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  10. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  11. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  12. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  13. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  14. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  15. 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
  16. 2020EasyQuant: Post-training Quantization via Scale Optimizationcited by 57
  17. 2020Training Binary Neural Networks through Learning with Noisy Supervisioncited by 39
  18. 2021Learning Frequency Domain Approximation for Binary Neural Networkscited by 38
  19. 2015Resiliency of Deep Neural Networks under Quantizationcited by 34
  20. 2019ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decompositioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 31; 2023: 57; 2024: 80; 2025: 74; 2026: 25.

274 papers.

Sort:

By area:

  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  3. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  4. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  5. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  6. 2023A Survey on Model Compression for Large Language Modelscited by 263
  7. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  8. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  9. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  10. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  11. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  12. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
38 more
  1. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  2. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  3. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  4. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  5. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  6. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  7. 2023DiffRate : Differentiable Compression Rate for Efficient Vision Transformerscited by 78
  8. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  9. 2024Test-Time Model Adaptation with Only Forward Passescited by 74
  10. 2022PD-Quant: Post-Training Quantization based on Prediction Difference Metriccited by 73
  11. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  12. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  13. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 64
  14. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  15. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  16. 2022Patch Similarity Aware Data-Free Quantization for Vision Transformerscited by 55
  17. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 55
  18. 2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 54
  19. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 49
  20. 2024Q-VLM: Post-training Quantization for Large Vision-Language Modelscited by 47
  21. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  22. 2023Towards Accurate Post-training Quantization for Diffusion Modelscited by 46
  23. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  24. 2024A Survey on Transformer Compressioncited by 44
  25. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  26. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  27. 2023A survey on efficient vision transformers: algorithms, techniques, and performance benchmarkingcited by 41
  28. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  29. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
  30. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  31. 2023CBQ: Cross-Block Quantization for Large Language Modelscited by 34
  32. 2023Dynamic Perceiver for Efficient Visual Recognitioncited by 33
  33. 2023TinySAM: Pushing the Envelope for Efficient Segment Anything Modelcited by 33
  34. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
  35. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  36. 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29
  37. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 28
  38. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28

50 most cited of 274 All 274 (JSON)