# TSPTQ-ViT: Two-scaled post-training quantization for vision transformer

arXiv: https://arxiv.org/abs/2305.12901 · HTML: https://mlsanity.com/paper/2305.12901/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [Towards Effective Low-bitwidth Convolutional Neural Networks](https://mlsanity.com/paper/1711.00205/) (2017)
- [Patch Similarity Aware Data-Free Quantization for Vision Transformers](https://mlsanity.com/paper/2203.02250/) (2022)
- [Q-ViT: Fully Differentiable Quantization for Vision Transformer](https://mlsanity.com/paper/2201.07703/) (2022)
- [Post-Training Sparsity-Aware Quantization](https://mlsanity.com/paper/2105.11010/) (2021)
- [Feature Map Transform Coding for Energy-Efficient CNN Inference](https://mlsanity.com/paper/1905.10830/) (2019)
- [TerViT: An Efficient Ternary Vision Transformer](https://mlsanity.com/paper/2201.08050/) (2022)
- [Transform-Based Feature Map Compression for CNN Inference](https://mlsanity.com/paper/2106.12850/) (2021)
- [Compression-aware Projection with Greedy Dimension Reduction for Convolutional Neural Network Activations](https://mlsanity.com/paper/2110.08828/) (2021)
- [Learnable Mixed-precision and Dimension Reduction Co-design for Low-storage Activation](https://mlsanity.com/paper/2207.07931/) (2022)

## Built on by (most cited first, 7 of 7)

- [TinySAM: Pushing the Envelope for Efficient Segment Anything Model](https://mlsanity.com/paper/2312.13789/) (2023)
- [Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies](https://mlsanity.com/paper/2503.02891/) (2025)
- [Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey](https://mlsanity.com/paper/2405.00314/) (2024)
- [MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2401.14895/) (2024)
- [Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivity](https://mlsanity.com/paper/2501.06357/) (2025)
- [Post-Training Quantization for Vision Mamba with k-Scaled Quantization and Reparameterization](https://mlsanity.com/paper/2501.16738/) (2025)
- [CoQMoE: Co-Designed Quantization and Computation Orchestration for Mixture-of-Experts Vision Transformer on FPGA](https://mlsanity.com/paper/2506.08496/) (2025)

## Abstract (arXiv)

Vision transformers (ViTs) have achieved remarkable performance in various computer vision tasks. However, intensive memory and computation requirements impede ViTs from running on resource-constrained edge devices. Due to the non-normally distributed values after Softmax and GeLU, post-training quantization on ViTs results in severe accuracy degradation. Moreover, conventional methods fail to address the high channel-wise variance in LayerNorm. To reduce the quantization loss and improve classification accuracy, we propose a two-scaled post-training quantization scheme for vision transformer (TSPTQ-ViT). We design the value-aware two-scaled scaling factors (V-2SF) specialized for post-Softmax and post-GeLU values, which leverage the bit sparsity in non-normal distribution to save bit-widths. In addition, the outlier-aware two-scaled scaling factors (O-2SF) are introduced to LayerNorm, alleviating the dominant impacts from outlier values. Our experimental results show that the proposed methods reach near-lossless accuracy drops (<0.5%) on the ImageNet classification task under 8-bit fully quantized ViTs.