mlsanityData to 7 Oct 2026

Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy

arXiv 2401.07250 ↗First posted 14 Jan 2024cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Recently, sharpness-aware minimization (SAM) has attracted much attention because of its surprising effectiveness in improving generalization performance. However, compared to stochastic gradient descent (SGD), it is more prone to getting stuck at the saddle points, which as a result may lead to performance degradation. To address this issue, we propose a simple renormalization strategy, dubbed Stable SAM (SSAM), so that the gradient norm of the descent step maintains the same as that of the ascent step. Our strategy is easy to implement and flexible enough to integrate with SAM and its variants, almost at no computational cost. With elementary tools from convex optimization and learning theory, we also conduct a theoretical analysis of sharpness-aware training, revealing that compared to SGD, the effectiveness of SAM is only assured in a limited regime of learning rate. In contrast, we show how SSAM extends this regime of learning rate and then it can consistently perform better than SAM with the minor modification. Finally, we demonstrate the improved performance of SSAM on several representative data sets and tasks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 61 of its 76 references that are in the map. First posted on arXiv 14 Jan 2024.

Builds on

61 of 76 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  5. 2016Wide Residual Networkscited by 3,057
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  11. 2019When Does Label Smoothing Help?cited by 985
  12. 2014Convex Optimization: Algorithms and Complexitycited by 569
38 more
  1. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  3. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  4. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  5. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  6. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  7. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  8. 2017Three Factors Influencing Minima in SGDcited by 270
  9. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  10. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  11. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  12. 2016Deep Pyramidal Residual Networkscited by 163
  13. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  14. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  15. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  16. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  17. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  18. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  19. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  20. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  21. 2022Sharpness-Aware Training for Freecited by 95
  22. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  23. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  24. 2022When Do Flat Minima Optimizers Work?cited by 78
  25. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  26. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  27. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  28. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
  29. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  30. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  31. 2023Benchmarking Neural Network Training Algorithmscited by 46
  32. 2019Loss Landscapes of Regularized Linear Autoencoderscited by 45
  33. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  34. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  35. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  36. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  37. 2023ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognitioncited by 31
  38. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shapingcited by 4
  2. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 3
  3. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  4. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  5. 2025Convergence of optimizers implies eigenvalues filtering at equilibriumcited by 0
  6. 2026Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementationcited by 0
  7. 2026Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimizationcited by 0
  8. 2026Sparsity-Adaptive Sharpness-Aware Minimizationcited by 0