mlsanityData to 7 Oct 2026

Patch-wise Mixed-Precision Quantization of Vision Transformer

arXiv 2305.06559 ↗First posted 11 May 2023cs.CVcited by 15

Not on the map. Why?

Abstract (arXiv)

As emerging hardware begins to support mixed bit-width arithmetic computation, mixed-precision quantization is widely used to reduce the complexity of neural networks. However, Vision Transformers (ViTs) require complex self-attention computation to guarantee the learning of powerful feature representations, which makes mixed-precision quantization of ViTs still challenging. In this paper, we propose a novel patch-wise mixed-precision quantization (PMQ) for efficient inference of ViTs. Specifically, we design a lightweight global metric, which is faster than existing methods, to measure the sensitivity of each component in ViTs to quantization errors. Moreover, we also introduce a pareto frontier approach to automatically allocate the optimal bit-precision according to the sensitivity. To further reduce the computational complexity of self-attention in inference stage, we propose a patch-wise module to reallocate bit-width of patches in each layer. Extensive experiments on the ImageNet dataset shows that our method greatly reduces the search cost and facilitates the application of mixed-precision quantization to ViTs.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 30 of its 34 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

30 of 34 references.

References · 30

  1. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  2. 2020End-to-End Object Detection with Transformerscited by 7,384
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  6. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  7. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  8. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  9. 2019Importance Estimation for Neural Network Pruningcited by 531
  10. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  11. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  12. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
18 more
  1. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  2. 2021Post-Training Quantization for Vision Transformercited by 274
  3. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  4. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  5. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  6. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 141
  7. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
  8. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 93
  9. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  10. 2020Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programmingcited by 89
  11. 2022MiniViT: Compressing Vision Transformers with Weight Multiplexingcited by 88
  12. 2019AutoQ: Automated Kernel-Wise Neural Network Quantizationcited by 60
  13. 2020HAWQV3: Dyadic Neural Network Quantizationcited by 59
  14. 2021OMPQ: Orthogonal Mixed Precision Quantizationcited by 37
  15. 2019Training Quantized Neural Networks with a Full-precision Auxiliary Modulecited by 35
  16. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  17. 2019Learning low-precision neural networks without Straight-Through Estimator(STE)cited by 15
  18. 2015Fixed-Point Performance Analysis of Recurrent Neural Networkscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 6; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024A Survey on Transformer Compressioncited by 44
  2. 2024EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Modelscited by 23
  3. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  4. 2024CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTscited by 17
  5. 2024On Efficient Variants of Segment Anything Model: A Surveycited by 16
  6. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 10
  7. 2025Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivitycited by 9
  8. 2024LRP-QViT: Mixed-Precision Vision Transformer Quantization via Layer-wise Relevance Propagationcited by 8
  9. 2023BinaryViT: Towards Efficient and Accurate Binary Vision Transformerscited by 7
  10. 2025Mix-QSAM: Mixed-Precision Quantization of the Segment Anything Modelcited by 4
  11. 2025MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspectivecited by 2
  12. 2024DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilationcited by 1
3 more
  1. 2025MLoRQ: Bridging Low-Rank and Quantization for Transformer Compressioncited by 1
  2. 2025Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformerscited by 1
  3. 2026Amortized-Precision Quantization for Early-Exit Vision Transformerscited by 0