mlsanityData to 7 Oct 2026

Instance-Aware Group Quantization for Vision Transformers

arXiv 2404.00928 ↗First posted 1 Apr 2024cs.CV cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Post-training quantization (PTQ) is an efficient model compression technique that quantizes a pretrained full-precision model using only a small calibration set of unlabeled samples without retraining. PTQ methods for convolutional neural networks (CNNs) provide quantization results comparable to full-precision counterparts. Directly applying them to vision transformers (ViTs), however, incurs severe performance degradation, mainly due to the differences in architectures between CNNs and ViTs. In particular, the distribution of activations for each channel vary drastically according to input instances, making PTQ methods for CNNs inappropriate for ViTs. To address this, we introduce instance-aware group quantization for ViTs (IGQ-ViT). To this end, we propose to split the channels of activation maps into multiple groups dynamically for each input instance, such that activations within each group share similar statistical properties. We also extend our scheme to quantize softmax attentions across tokens. In addition, the number of groups for each layer is adjusted to minimize the discrepancies between predictions from quantized and full-precision models, under a bit-operation (BOP) constraint. We show extensive experimental results on image classification, object detection, and instance segmentation, with various transformer architectures, demonstrating the effectiveness of our approach.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

35 of 44 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2017Mask R-CNNcited by 7,259
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2016Layer Normalizationcited by 5,579
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
23 more
  1. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  2. 2017Cascade R-CNN: Delving into High Quality Object Detectioncited by 1,275
  3. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  4. 2019Learned Step Size Quantizationcited by 604
  5. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  6. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
  7. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  8. 2021Post-Training Quantization for Vision Transformercited by 274
  9. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 211
  10. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  11. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  12. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  13. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  14. 2019Loss Aware Post-training Quantizationcited by 94
  15. 2021Network Quantization with Element-wise Gradient Scalingcited by 82
  16. 2020Rethinking Differentiable Search for Mixed-Precision Neural Networkscited by 79
  17. 2020Post-Training Piecewise Linear Quantization for Deep Neural Networkscited by 77
  18. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 76
  19. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  20. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  21. 2020PROFIT: A Novel Training Method for sub-4-bit MobileNet Modelscited by 50
  22. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
  23. 2021Distance-aware Quantizationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 12; 2026: 8.

23 papers.

Sort:

By area:

  1. 2024Q-DiT: Accurate Post-Training Quantization for Diffusion Transformerscited by 91
  2. 2024Quantization without Tearscited by 24
  3. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  4. 2025APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformerscited by 17
  5. 2025PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancementcited by 17
  6. 2025FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximationcited by 16
  7. 2025Accelerating Vision Transformers with Adaptive Patch Sizescited by 12
  8. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  9. 2025Diffusion Model Quantization: A Reviewcited by 6
  10. 2025SL-ACC: A Communication-Efficient Split Learning Framework with Adaptive Channel-wise Compressioncited by 6
  11. 2025AHCQ-SAM: Toward Accurate and Hardware-Compatible Post-Training Segment Anything Model Quantizationcited by 3
  12. 2025Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracycited by 3
11 more
  1. 2025AccuQuant: Simulating Multiple Denoising Steps for Quantizing Diffusion Modelscited by 3
  2. 2025LampQ: Towards Accurate Layer-wise Mixed Precision Quantization for Vision Transformerscited by 2
  3. 2026Nonlinear Bipolar Compensation: Handling Outliers in Post-Training Quantizationcited by 1
  4. 2025Assessing the Potential for Catastrophic Failure in Dynamic Post-Training Quantizationcited by 0
  5. 2026When W4A4 Breaks Camouflaged Object Detection: Token-Group Dual-Constraint Activation Quantizationcited by 0
  6. 2026Toward Multi-Domain and Long-Tailed Quantization via Feature Alignment and Scalingcited by 0
  7. 2026Shift-and-Sum Quantization for Visual Autoregressive Modelscited by 0
  8. 2026ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformerscited by 0
  9. 2026QuaSAR: Quantization Compensation via Stable Activation-Aware Rank Truncationcited by 0
  10. 2026Channel-Wise and Token-Aware Post-Training Quantization for Visual State Space Dualitycited by 0
  11. 2026GLF-Q: Global-Local Feature-based Quantization for Vision Transformerscited by 0