mlsanityData to 7 Oct 2026

FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer

arXiv 2111.13824 ↗First posted 27 Nov 2021cs.CVcited by 148

Not on the map. Why?

Abstract (arXiv)

Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.

Key facts. Cited by 148 papers in the map (data through 7 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 27 Nov 2021.

Builds on

20 of 28 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Layer Normalizationcited by 5,579
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  10. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
8 more
  1. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  2. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  3. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  4. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  5. 2021Post-Training Quantization for Vision Transformercited by 274
  6. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  7. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 185
  8. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 18; 2024: 56; 2025: 40; 2026: 28.

148 papers.

Sort:

By area:

  1. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 167
  2. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  3. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  4. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  5. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  6. 2023EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Modelscited by 77
  7. 2023TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Modelscited by 67
  8. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  9. 2023Which Tokens to Use? Investigating Token Reduction in Vision Transformerscited by 59
  10. 2024A Survey on Transformer Compressioncited by 44
  11. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  12. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
38 more
  1. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  2. 2024PTQ4SAM: Post-Training Quantization for Segment Anythingcited by 32
  3. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  4. 2024AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizercited by 28
  5. 2023Stitchable Neural Networkscited by 26
  6. 2025MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methodscited by 26
  7. 2024Quantization without Tearscited by 24
  8. 2025Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategiescited by 24
  9. 2023Q-DETR: An Efficient Low-Bit Quantized Detection Transformercited by 23
  10. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  11. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  12. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantizationcited by 19
  13. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Modelscited by 19
  14. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformerscited by 19
  15. 2025A Survey on Efficient Vision-Language Modelscited by 19
  16. 2023Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformerscited by 18
  17. 2023BinaryViT: Pushing Binary Vision Transformers Towards Convolutional Modelscited by 17
  18. 2024CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTscited by 17
  19. 2025APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformerscited by 17
  20. 2025FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximationcited by 16
  21. 2024HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantizationcited by 15
  22. 2023Quantized Feature Distillation for Network Quantizationcited by 13
  23. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
  24. 2024DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothingcited by 13
  25. 2024Agglomerative Token Clusteringcited by 13
  26. 2024COMQ: A Backpropagation-Free Algorithm for Post-Training Quantizationcited by 12
  27. 2024BinaryDM: Accurate Weight Binarization for Efficient Diffusion Modelscited by 11
  28. 2025DGQ: Distribution-Aware Group Quantization for Text-to-Image Diffusion Modelscited by 11
  29. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  30. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformercited by 10
  31. 2024RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterizationcited by 10
  32. 2024PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inferencecited by 10
  33. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 10
  34. 2025QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operationscited by 10
  35. 2024Adaptive Layer Selection for Efficient Vision Transformer Fine-Tuningcited by 9
  36. 2025Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivitycited by 9
  37. 2024LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagationcited by 8
  38. 2023A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantizationcited by 7

50 most cited of 148 All 148 (JSON)