mlsanityData to 7 Oct 2026

Mixed Non-linear Quantization for Vision Transformers

arXiv 2407.18437 ↗First posted 26 Jul 2024cs.CV cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

The majority of quantization methods have been proposed to reduce the model size of Vision Transformers, yet most of them have overlooked the quantization of non-linear operations. Only a few works have addressed quantization for non-linear operations, but they applied a single quantization method across all non-linear operations. We believe that this can be further improved by employing a different quantization method for each non-linear operation. Therefore, to assign the most error-minimizing quantization method from the known methods to each non-linear layer, we propose a mixed non-linear quantization that considers layer-wise quantization sensitivity measured by SQNR difference metric. The results show that our method outperforms I-BERT, FQ-ViT, and I-ViT in both 8-bit and 6-bit settings for ViT, DeiT, and Swin models by an average of 0.6%p and 19.6%p, respectively. Our method outperforms I-BERT and I-ViT by 0.6%p and 20.8%p, respectively, when training time is limited. We plan to release our code at https://gitlab.com/ones-ai/mixed-non-linear-quantization.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 26 Jul 2024.

Builds on

26 of 33 references.

References · 26

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  8. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
  9. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  10. 2021Post-Training Quantization for Vision Transformercited by 274
  11. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  12. 2021I-BERT: Integer-only BERT Quantizationcited by 250
14 more
  1. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
  2. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  3. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformerscited by 134
  4. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  5. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  6. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  7. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 55
  8. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 36
  9. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  10. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantizationcited by 19
  11. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  12. 2023Bi-ViT: Pushing the Limit of Vision Transformer Quantizationcited by 13
  13. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
  14. 2024ViT-1.58b: Mobile Vision Transformers in the 1-bit Eracited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025I-Segmenter: Integer-Only Vision Transformer for Efficient Semantic Segmentationcited by 1
  2. 2025IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformerscited by 1
  3. 2026QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attentioncited by 0