mlsanityData to 7 Oct 2026

Towards Understanding Sharpness-Aware Minimization

arXiv 2206.06232 ↗First posted 13 Jun 2022cs.LGcited by 145

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based on a PAC-Bayes generalization bound and the idea of convergence to flat minima are incomplete. Moreover, there are no explanations for the success of using m-sharpness in SAM which has been shown as essential for generalization. To better understand this aspect of SAM, we theoretically analyze its implicit bias for diagonal linear networks. We prove that SAM always chooses a solution that enjoys better generalization properties than standard gradient descent for a certain class of problems, and this effect is amplified by using m-sharpness. We further study the properties of the implicit bias on non-linear networks empirically, where we show that fine-tuning a standard model with SAM can lead to significant generalization improvements. Finally, we provide convergence results of SAM for non-convex objectives when used with stochastic gradients. We illustrate these results empirically for deep networks and discuss their relation to the generalization behavior of SAM. The code of our experiments is available at https://github.com/tml-epfl/understanding-sam.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 32 of its 48 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

32 of 48 references.

References · 32

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2018Group Normalizationcited by 1,514
  5. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  9. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  10. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
20 more
  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  2. 2020Overfitting in adversarially robust deep learningcited by 504
  3. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  4. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  5. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  6. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  7. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  9. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  10. 2019SGD: General Analysis and Improved Ratescited by 206
  11. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  12. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  13. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  14. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  15. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  16. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  17. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  18. 2018SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learningcited by 25
  19. 2020Extrapolation for Large-batch Training in Deep Learningcited by 25
  20. 2019Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGDcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 43; 2024: 43; 2025: 34; 2026: 17.

145 papers.

Sort:

By area:

  1. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  2. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  3. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  4. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  5. 2024Friendly Sharpness-Aware Minimizationcited by 52
  6. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  7. 2024Deep Networks Always Grok and Here is Whycited by 49
  8. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  9. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  10. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  11. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  12. 2024Make Continual Learning Stronger via C-Flatcited by 35
38 more
  1. 2024Does SGD really happen in tiny subspaces?cited by 35
  2. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  3. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  4. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  5. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  6. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  7. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  8. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  9. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  10. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  11. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  12. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  13. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  14. 2024Why is SAM Robust to Label Noise?cited by 21
  15. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  16. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  17. 2023FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learningcited by 20
  18. 2022Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Biascited by 16
  19. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  20. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  21. 2023Stability Analysis of Sharpness-Aware Minimizationcited by 14
  22. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  23. 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
  24. 2024Implicit Optimization Bias of Next-Token Prediction in Linear Modelscited by 14
  25. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  26. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  27. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  28. 2023Chasing Fairness Under Distribution Shift: A Model Weight Perturbation Approachcited by 12
  29. 2023Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalizationcited by 12
  30. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
  31. 2025A Theoretical Framework for Grokking: Interpolation followed by Riemannian Norm Minimisationcited by 11
  32. 2023Approximately Bayes-Optimal Pseudo Label Selectioncited by 10
  33. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  34. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
  35. 2025Sharpness-Aware Minimization: General Analysis and Improved Ratescited by 10
  36. 2023Exploring the Effect of Multi-step Ascent in Sharpness-Aware Minimizationcited by 9
  37. 2023Multi-Dataset Co-Training with Sharpness-Aware Optimization for Audio Anti-spoofingcited by 9
  38. 2024Unveiling Privacy, Memorization, and Input Curvature Linkscited by 9

50 most cited of 145 All 145 (JSON)