# I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantization

arXiv: https://arxiv.org/abs/2311.10126 · HTML: https://mlsanity.com/paper/2311.10126/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 61 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 30 of 52)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [SwinIR: Image Restoration Using Swin Transformer](https://mlsanity.com/paper/2108.10257/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization](https://mlsanity.com/paper/2203.05740/) (2022)
- [PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization](https://mlsanity.com/paper/2111.12293/) (2021)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer](https://mlsanity.com/paper/2111.13824/) (2021)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)

## Built on by (most cited first, 19 of 19)

- [Flow caching for autoregressive video generation](https://mlsanity.com/paper/2602.10825/) (2026)
- [DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers](https://mlsanity.com/paper/2408.03291/) (2024)
- [APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers](https://mlsanity.com/paper/2504.02508/) (2025)
- [FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation](https://mlsanity.com/paper/2506.11543/) (2025)
- [Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey](https://mlsanity.com/paper/2405.00314/) (2024)
- [LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation](https://mlsanity.com/paper/2508.03485/) (2025)
- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers](https://mlsanity.com/paper/2407.02763/) (2024)
- [Motion-Aware Caching for Efficient Autoregressive Video Generation](https://mlsanity.com/paper/2605.01725/) (2026)
- [Mixed Non-linear Quantization for Vision Transformers](https://mlsanity.com/paper/2407.18437/) (2024)
- [Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers](https://mlsanity.com/paper/2412.16553/) (2024)
- [AHCQ-SAM: Toward Accurate and Hardware-Compatible Post-Training Segment Anything Model Quantization](https://mlsanity.com/paper/2503.03088/) (2025)
- [Pack-PTQ: Advancing Post-training Quantization of Neural Networks by Pack-wise Reconstruction](https://mlsanity.com/paper/2505.00259/) (2025)
- [Progressive Fine-to-Coarse Reconstruction for Accurate Low-Bit Post-Training Quantization in Vision Transformers](https://mlsanity.com/paper/2412.14633/) (2024)
- [ALGOGEN: Tool-Generated Verifiable Traces for Reliable Algorithm Visualization](https://mlsanity.com/paper/2605.12159/) (2026)
- [MARR: Module-Adaptive Residual Reconstruction for Low-Bit Post-Training Quantization](https://mlsanity.com/paper/2605.17997/) (2026)
- [ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers](https://mlsanity.com/paper/2606.21947/) (2026)
- [GLF-Q: Global-Local Feature-based Quantization for Vision Transformers](https://mlsanity.com/paper/2609.34564/) (2026)
- [STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization](https://mlsanity.com/paper/2609.38169/) (2026)

## Abstract (arXiv)

Albeit the scalable performance of vision transformers (ViTs), the dense computational costs (training & inference) undermine their position in industrial applications. Post-training quantization (PTQ), tuning ViTs with a tiny dataset and running in a low-bit format, well addresses the cost issue but unluckily bears more performance drops in lower-bit cases. In this paper, we introduce I&S-ViT, a novel method that regulates the PTQ of ViTs in an inclusive and stable fashion. I&S-ViT first identifies two issues in the PTQ of ViTs: (1) Quantization inefficiency in the prevalent log2 quantizer for post-Softmax activations; (2) Rugged and magnified loss landscape in coarse-grained quantization granularity for post-LayerNorm activations. Then, I&S-ViT addresses these issues by introducing: (1) A novel shift-uniform-log2 quantizer (SULQ) that incorporates a shift mechanism followed by uniform quantization to achieve both an inclusive domain representation and accurate distribution approximation; (2) A three-stage smooth optimization strategy (SOS) that amalgamates the strengths of channel-wise and layer-wise quantization to enable stable learning. Comprehensive evaluations across diverse vision tasks validate I&S-ViT' superiority over existing PTQ of ViTs methods, particularly in low-bit scenarios. For instance, I&S-ViT elevates the performance of 3-bit ViT-B by an impressive 50.68%.