mlsanityData to 14 Oct 2026

Sharpness-Aware Minimization: General Analysis and Improved Rates

arXiv 2503.02225 ↗First posted 4 Mar 2025math.OC cs.LG stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) has emerged as a powerful method for improving generalization in machine learning models by minimizing the sharpness of the loss landscape. However, despite its success, several important questions regarding the convergence properties of SAM in non-convex settings are still open, including the benefits of using normalization in the update rule, the dependence of the analysis on the restrictive bounded variance assumption, and the convergence guarantees under different sampling strategies. To address these questions, in this paper, we provide a unified analysis of SAM and its unnormalized variant (USAM) under one single flexible update rule (Unified SAM), and we present convergence results of the new algorithm under a relaxed and more natural assumption on the stochastic noise. Our analysis provides convergence guarantees for SAM under different step size selections for non-convex problems and functions that satisfy the Polyak-Lojasiewicz (PL) condition (a non-convex generalization of strongly convex functions). The proposed theory holds under the arbitrary sampling paradigm, which includes importance sampling as special case, allowing us to analyze variants of SAM that were never explicitly considered in the literature. Experiments validate the theoretical findings and further demonstrate the practical effectiveness of Unified SAM in training deep neural networks for image classification tasks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

39 of 48 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Wide Residual Networkscited by 3,070
  3. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  10. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  11. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  12. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
27 more
  1. 2019SGD: General Analysis and Improved Ratescited by 207
  2. 2020Bag of Tricks for Adversarial Trainingcited by 200
  3. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  4. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  5. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  6. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 142
  7. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  8. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  9. 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 67
  10. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  11. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  12. 2019Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptionscited by 48
  13. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  14. 2022Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methodscited by 37
  15. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  16. 2020Unified Analysis of Stochastic Gradient Methods for Composite Convex and Smooth Optimizationcited by 36
  17. 2020Stochastic Hamiltonian Gradient Methods for Smooth Gamescited by 32
  18. 2021Stochastic Extragradient: General Analysis and Improved Ratescited by 31
  19. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  20. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  21. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  22. 2023Single-Call Stochastic Extragradient Methods for Structured Non-monotone Variational Inequalities: Improved Analysis under Weaker Conditionscited by 14
  23. 2024Fundamental Convergence Analysis of Sharpness-Aware Minimizationcited by 14
  24. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  25. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  26. 2025Avoiding spurious sharpness minimization broadens applicability of SAMcited by 8
  27. 2024Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Ratecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  2. 2026Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Schedulercited by 1
  3. 20265% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuningcited by 1
  4. 2025DGSAM: Domain Generalization via Individual Sharpness-Aware Minimizationcited by 0
  5. 2025Sharpness-Aware Minimization with Z-Score Gradient Filteringcited by 0
  6. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  7. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  8. 2026X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correctioncited by 0
  9. 2026Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learningcited by 0
  10. 2026On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Boundscited by 0