mlsanityData to 14 Oct 2026

Stabilizing Quantization-Aware Training by Implicit-Regularization on Hessian Matrix

arXiv 2503.11159 ↗First posted 14 Mar 2025cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Quantization-Aware Training (QAT) is one of the prevailing neural network compression solutions. However, its stability has been challenged for yielding deteriorating performances as the quantization error is inevitable. We find that the sharp landscape of loss, which leads to a dramatic performance drop, is an essential factor that causes instability. Theoretically, we have discovered that the perturbations in the feature would bring a flat local minima. However, simply adding perturbations into either weight or feature empirically deteriorates the performance of the Full Precision (FP) model. In this paper, we propose Feature-Perturbed Quantization (FPQ) to stochastically perturb the feature and employ the feature distillation method to the quantized model. Our method generalizes well to different network architectures and various QAT methods. Furthermore, we mathematically show that FPQ implicitly regularizes the Hessian norm, which calibrates the smoothness of a loss landscape. Extensive experiments demonstrate that our approach significantly outperforms the current State-Of-The-Art (SOTA) QAT methods and even the FP counterparts.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 42 of its 47 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

42 of 47 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  8. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  11. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  12. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
30 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  2. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  3. 2019Learned Step Size Quantizationcited by 628
  4. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  5. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  6. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  7. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  8. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  9. 2019Adversarial Examples Improve Image Recognitioncited by 286
  10. 2023Q-Diffusion: Quantizing Diffusion Modelscited by 273
  11. 2020LSQ+: Improving low-bit quantization through learnable offsets and better initializationcited by 189
  12. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  13. 2023PTQD: Accurate Post-Training Quantization for Diffusion Modelscited by 170
  14. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  15. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 119
  16. 2020Stabilizing Differentiable Architecture Search via Perturbation-based Regularizationcited by 102
  17. 2019Defensive Quantization: When Efficiency Meets Robustnesscited by 95
  18. 2021Network Quantization with Element-wise Gradient Scalingcited by 87
  19. 2023Temporal Dynamic Quantization for Diffusion Modelscited by 81
  20. 2023TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Modelscited by 69
  21. 2021Differentiable Model Compression via Pseudo Quantization Noisecited by 45
  22. 2022PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformerscited by 42
  23. 2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 40
  24. 2018Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?cited by 39
  25. 2024BitsFusion: 1.99 bits Weight Quantization of Diffusion Modelcited by 37
  26. 2022NIPQ: Noise proxy-based Integrated Pseudo-Quantizationcited by 35
  27. 2018L2-Nonexpansive Neural Networkscited by 21
  28. 2023Resilient Binary Neural Networkcited by 13
  29. 2024Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimizationcited by 5
  30. 2024In-Distribution Consistency Regularization Improves the Generalization of Quantization-Aware Trainingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunitiescited by 11
  2. 2026Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Compute-Optimal Quantization-Aware Training
  2. 2026StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
  3. 2026Quantization-Aware Regularizers for Deep Neural Networks Compression
  4. 2023On Uniform Scalar Quantization for Learned Image Compression
  5. 2022Error-aware Quantization through Noise Tempering
  6. 2021Sharpness-aware Quantization for Deep Neural Networks
  7. 2026HAMP-LIC: Hessian-Aware Mixed-Precision Post-Training Quantization for Learned Image Compression
  8. 2022QFT: Post-training quantization via fast joint finetuning of all degrees of freedom