mlsanityData to 14 Oct 2026

Oscillation-free Quantization for Low-bit Vision Transformers

arXiv 2302.02210 ↗First posted 4 Feb 2023cs.CV cs.AI cs.ARcited by 57

Not on the map. Why?

Abstract (arXiv)

Weight oscillation is an undesirable side effect of quantization-aware training, in which quantized weights frequently jump between two quantized levels, resulting in training instability and a sub-optimal final model. We discover that the learnable scaling factor, a widely-used textitde facto setting in quantization aggravates weight oscillation. In this study, we investigate the connection between the learnable scaling factor and quantized weight oscillation and use ViT as a case driver to illustrate the findings and remedies. In addition, we also found that the interdependence between quantized weights in textitquery and textitkey of a self-attention layer makes ViT vulnerable to oscillation. We, therefore, propose three techniques accordingly: statistical weight quantization (rm StatsQ) to improve quantization robustness compared to the prevalent learnable-scale-based method; confidence-guided annealing (rm CGA) that freezes the weights with textithigh confidence and calms the oscillating weights; and textitquery-textitkey reparameterization (rm QKR) to resolve the query-key intertwined oscillation and mitigate the resulting gradient misestimation. Extensive experiments demonstrate that these proposed techniques successfully abate weight oscillation and consistently achieve substantial accuracy improvement on ImageNet. Specifically, our 2-bit DeiT-T/DeiT-S algorithms outperform the previous state-of-the-art by 9.8% and 7.7%, respectively. Code and models are available at: https://github.com/nbasyl/OFQ.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 29 of its 34 references that are in the map. First posted on arXiv 4 Feb 2023.

Builds on

29 of 34 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  7. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  9. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  10. 2019Learned Step Size Quantizationcited by 628
  11. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  12. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
17 more
  1. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  2. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  3. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  4. 2021Post-Training Quantization for Vision Transformercited by 278
  5. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  6. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 238
  7. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 235
  8. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 227
  9. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 148
  10. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  11. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  12. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 99
  13. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  14. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 79
  15. 2019Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimizationcited by 60
  16. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  17. 2022Q-ViT: Fully Differentiable Quantization for Vision Transformercited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 15; 2025: 24; 2026: 11.

57 papers.

Sort:

By area:

  1. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  2. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  3. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 79
  4. 2023Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMscited by 50
  5. 2024A Survey on Transformer Compressioncited by 46
  6. 2024Towards Accurate Post-Training Quantization of Vision Transformers via Error Reductioncited by 23
  7. 2025Oscillation-Reduced MXFP4 Training for Vision Transformerscited by 21
  8. 2023I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantizationcited by 19
  9. 2024DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformerscited by 19
  10. 2023Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformerscited by 18
  11. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  12. 2025TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Controlcited by 14
38 more
  1. 2024Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Surveycited by 13
  2. 2025Low-bit Model Quantization for Deep Neural Networks: A Surveycited by 11
  3. 2025Rapid Salient Object Detection with Difference Convolutional Neural Networkscited by 10
  4. 2025Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivitycited by 9
  5. 2025GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformerscited by 8
  6. 2024EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximationcited by 7
  7. 2023Reducing the Side-Effects of Oscillations in Training of Quantized YOLO Networkscited by 6
  8. 2024Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuningcited by 6
  9. 2025BHViT: Binarized Hybrid Vision Transformercited by 5
  10. 2025Improving the Straight-Through Estimator with Zeroth-Order Informationcited by 5
  11. 2024Mixed Non-linear Quantization for Vision Transformerscited by 3
  12. 2024Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformerscited by 3
  13. 2025AHCQ-SAM: Toward Accurate and Hardware-Compatible Post-Training Segment Anything Model Quantizationcited by 3
  14. 2025MixA-Q: Revisiting Activation Sparsity for Vision Transformers from a Mixed-Precision Quantization Perspectivecited by 3
  15. 2025Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Samplingcited by 3
  16. 2023Efficient and Robust Quantization-aware Training via Adaptive Coreset Selectioncited by 2
  17. 2024Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformerscited by 2
  18. 2024GAQAT: gradient-adaptive quantization-aware training for domain generalizationcited by 2
  19. 2025Enhancing Quantization-Aware Training on Edge Devices via Relative Entropy Coreset Selection and Cascaded Layer Correctioncited by 2
  20. 2025MSQ: Memory-Efficient Bit Sparsification Quantizationcited by 2
  21. 2025Quantization Meets OOD: Generalizable Quantization-aware Training from a Flatness Perspectivecited by 2
  22. 2025Oscillations Make Neural Networks Robust to Quantizationcited by 1
  23. 2025Q-SAM2: Accurate Quantization for Segment Anything Model 2cited by 1
  24. 2026SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Modelscited by 1
  25. 2026Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decaycited by 1
  26. 2026SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Featurescited by 1
  27. 2024OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learningcited by 0
  28. 2024Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Modelscited by 0
  29. 2024Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teachercited by 0
  30. 2025Q&C: When Quantization Meets Cache in Efficient Image Generationcited by 0
  31. 2025SSVQ: Unleashing the Potential of Vector Quantization with Sign-Splittingcited by 0
  32. 2025NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devicescited by 0
  33. 2025APSQ: Additive Partial Sum Quantization with Algorithm-Hardware Co-Designcited by 0
  34. 2025Binary Quadratic Quantization: Beyond First-Order Quantization for Real-Valued Matrix Compressioncited by 0
  35. 2025Extreme Model Compression with Structured Sparsity at Low Precisioncited by 0
  36. 2025D4C: Data-Free Quantization for Contrastive Language-Image Pre-training Modelscited by 0
  37. 2025SignRoundV2: Toward Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMscited by 0
  38. 2026pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Trainingcited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers
  2. 2026ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers
  3. 2023Bi-ViT: Pushing the Limit of Vision Transformer Quantization
  4. 2026Zero-Shot Quantization via Weight-Space Arithmetic
  5. 2024MPTQ-ViT: Mixed-Precision Post-Training Quantization for Vision Transformer
  6. 2023TSPTQ-ViT: Two-scaled post-training quantization for vision transformer
  7. 2024Scheduling Weight Transitions for Quantization-Aware Training
  8. 2022RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers