mlsanityData to 14 Oct 2026

Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformers

arXiv 2308.10814 ↗First posted 21 Aug 2023cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Quantization scale and bit-width are the most important parameters when considering how to quantize a neural network. Prior work focuses on optimizing quantization scales in a global manner through gradient methods (gradient descent & Hessian analysis). Yet, when applying perturbations to quantization scales, we observe a very jagged, highly non-smooth test loss landscape. In fact, small perturbations in quantization scale can greatly affect accuracy, yielding a 0.5-0.8% accuracy boost in 4-bit quantized vision transformers (ViTs). In this regime, gradient methods break down, since they cannot reliably reach local minima. In our work, dubbed Evol-Q, we use evolutionary search to effectively traverse the non-smooth landscape. Additionally, we propose using an infoNCE loss, which not only helps combat overfitting on the small calibration dataset (1,000 images) but also makes traversing such a highly non-smooth surface easier. Evol-Q improves the top-1 accuracy of a fully quantized ViT-Base by 10.30%, 0.78%, and 0.15% for 3-bit, 4-bit, and 8-bit weight quantization levels. Extensive experiments on a variety of CNN and ViT architectures further demonstrate its robustness in extreme quantization scenarios. Our code is available at https://github.com/enyac-group/evol-q

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

28 of 36 references.

References · 28

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  9. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  10. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  11. 2021Post-Training Quantization for Vision Transformercited by 278
  12. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
16 more
  1. 2019Low-bit Quantization of Neural Networks for Efficient Inferencecited by 193
  2. 2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 179
  3. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  4. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  5. 2019Loss Aware Post-training Quantizationcited by 97
  6. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  7. 2020Data-Free Network Quantization With Adversarial Knowledge Distillationcited by 80
  8. 2020EasyQuant: Post-training Quantization via Scale Optimizationcited by 57
  9. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  10. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  11. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 37
  12. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 29
  13. 2022Neural Network Quantization with AI Model Efficiency Toolkit (AIMET)cited by 24
  14. 2022Network Binarization via Contrastive Learningcited by 12
  15. 2022TerViT: An Efficient Ternary Vision Transformercited by 11
  16. 2022Synergistic Self-supervised and Quantization Learningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 5; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 79
  2. 2024A Survey on Transformer Compressioncited by 46
  3. 2024AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizercited by 29
  4. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  5. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantizationcited by 19
  6. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformerscited by 19
  7. 2024CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTscited by 17
  8. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
  9. 2025QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Modelscited by 11
  10. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  11. 2024Scaling Laws for Post Training Quantized Large Language Modelscited by 4
  12. 2024Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformerscited by 3
6 more
  1. 2025AHCQ-SAM: Toward Accurate and Hardware-Compatible Post-Training Segment Anything Model Quantizationcited by 3
  2. 2025MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspectivecited by 3
  3. 2026Zero-Shot Quantization via Weight-Space Arithmeticcited by 2
  4. 2025AIRCHITECT v2: Learning the Hardware Accelerator Design Space through Unified Representationscited by 1
  5. 2024Curvature in the Looking-Glass: Optimal Methods to Exploit Curvature of Expectation in the Loss Landscapecited by 0
  6. 2026MixFrag: Fragility-Guided Mixed-Precision Post-Training Quantization for Vision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022CPT-V: A Contrastive Approach to Post-Training Quantization of Vision Transformers
  2. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer
  3. 2026ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers
  4. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision
  5. 2024ViT-1.58b: Mobile Vision Transformers in the 1-bit Era
  6. 2024ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers
  7. 2025GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
  8. 2025LampQ: Towards Accurate Layer-wise Mixed Precision Quantization for Vision Transformers