mlsanityData to 7 Oct 2026

The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minima

arXiv 2210.01513 ↗First posted 4 Oct 2022cs.LG math.OC stat.MLcited by 51

Not on the map. Why?

Abstract (arXiv)

We consider Sharpness-Aware Minimization (SAM), a gradient-based optimization method for deep networks that has exhibited performance improvements on image and language prediction problems. We show that when SAM is applied with a convex quadratic objective, for most random initializations it converges to a cycle that oscillates between either side of the minimum in the direction with the largest curvature, and we provide bounds on the rate of convergence. In the non-quadratic case, we show that such oscillations effectively perform gradient descent, with a smaller step-size, on the spectral norm of the Hessian. In such cases, SAM's update may be regarded as a third derivative -- the derivative of the Hessian in the leading eigenvector direction -- that encourages drift toward wider minima.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

20 of 25 references.

References · 20

  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  4. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  5. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  6. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  7. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  8. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  9. 2021Deep learning: a statistical viewpointcited by 239
  10. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 183
  11. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  12. 2020Implicit Gradient Regularizationcited by 145
8 more
  1. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  2. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  3. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
  4. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  5. 2022Understanding the unstable convergence of gradient descentcited by 84
  6. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  7. 2019Sharp Analysis for Nonconvex SGD Escaping from Saddle Pointscited by 40
  8. 2022On the Benefits of Large Learning Rates for Kernel Methodscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 13; 2025: 14; 2026: 7.

51 papers.

Sort:

By area:

  1. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  2. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  3. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  4. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  5. 2024Does SGD really happen in tiny subspaces?cited by 35
  6. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  7. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  8. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  9. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  10. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  11. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  12. 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
38 more
  1. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  2. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  3. 2023Stability Analysis of Sharpness-Aware Minimizationcited by 14
  4. 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
  5. 2024Learning Associative Memories with Gradient Descentcited by 14
  6. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  7. 2023How to escape sharp minima with random perturbationscited by 13
  8. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  9. 2025Sharpness-Aware Minimization: General Analysis and Improved Ratescited by 10
  10. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  11. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  12. 2024Tilted Sharpness-Aware Minimizationcited by 6
  13. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  14. 2023On Statistical Properties of Sharpness-Aware Minimization: Provable Guaranteescited by 5
  15. 2024Simplicity Bias via Global Convergence of Sharpness Minimizationcited by 4
  16. 2024Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalizationcited by 3
  17. 2025Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learningcited by 3
  18. 2025Convergence Rates for Gradient Descent on the Edge of Stability in Overparametrised Least Squarescited by 3
  19. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 3
  20. 2024A Precise Characterization of SGD Stability Using Loss Surface Geometrycited by 2
  21. 2025Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Modelscited by 2
  22. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  23. 2025VASSO: Variance Suppression for Sharpness-Aware Minimizationcited by 1
  24. 2026Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMscited by 1
  25. 2026When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Studycited by 1
  26. 2026Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuationscited by 1
  27. 2023A simple connection from loss flatness to compressed neural representationscited by 0
  28. 2024Beyond Single-Model Views for Deep Learning: Optimization versus Generalizability of Stochastic Optimization Algorithmscited by 0
  29. 2024Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamicscited by 0
  30. 2024Understanding SAM's Robustness to Noisy Labels through Gradient Down-weightingcited by 0
  31. 2025Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Modelscited by 0
  32. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  33. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  34. 2025Zeroth-Order Sharpness-Aware Learning with Exponential Tiltingcited by 0
  35. 2025The Interplay of Statistics and Noisy Optimization: Learning Linear Predictors with Random Data Weightscited by 0
  36. 2026X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correctioncited by 0
  37. 2026Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMscited by 0
  38. 2026Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementationcited by 0

50 most cited of 51 All 51 (JSON)