mlsanityData to 7 Oct 2026

Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimization

arXiv 2403.08821 ↗First posted 24 Feb 2024cs.CV cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware Minimization (SAM) has been proposed recently to improve model generalization ability. However, SAM calculates the gradient twice in each optimization step, thereby doubling the computation costs compared to stochastic gradient descent (SGD). In this paper, we propose a simple yet efficient sampling method to significantly accelerate SAM. Concretely, we discover that the gradient of SAM is a combination of the gradient of SGD and the Projection of the Second-order gradient matrix onto the First-order gradient (PSF). PSF exhibits a gradually increasing frequency of change during the training process. To leverage this observation, we propose an adaptive sampling method based on the variation of PSF, and we reuse the sampled PSF for non-sampling iterations. Extensive empirical results illustrate that the proposed method achieved state-of-the-art accuracies comparable to SAM on diverse network architectures.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 24 Feb 2024.

Builds on

27 of 36 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2016Wide Residual Networkscited by 3,057
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2019Learned Step Size Quantizationcited by 604
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  12. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
15 more
  1. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  2. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  3. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  4. 2016Deep Pyramidal Residual Networkscited by 163
  5. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  6. 2022Understanding The Robustness in Vision Transformerscited by 144
  7. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  8. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  9. 2022Overcoming Oscillations in Quantization-Aware Trainingcited by 116
  10. 2022Sharpness-Aware Training for Freecited by 95
  11. 2020On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Themcited by 59
  12. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  13. 2019Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margincited by 56
  14. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  15. 2020Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruptioncited by 26

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimizationcited by 5