mlsanityData to 14 Oct 2026

An SDE for Modeling SAM: Theory and Insights

arXiv 2301.08203 ↗First posted 19 Jan 2023cs.LG math.OCcited by 22

Not on the map. Why?

Abstract (arXiv)

We study the SAM (Sharpness-Aware Minimization) optimizer which has recently attracted a lot of interest due to its increased performance over more classical variants of stochastic gradient descent. Our main contribution is the derivation of continuous-time models (in the form of SDEs) for SAM and two of its variants, both for the full-batch and mini-batch settings. We demonstrate that these SDEs are rigorous approximations of the real discrete-time algorithms (in a weak sense, scaling linearly with the learning rate). Using these models, we then offer an explanation of why SAM prefers flat minima over sharp ones~--~by showing that it minimizes an implicitly regularized loss with a Hessian-dependent noise structure. Finally, we prove that SAM is attracted to saddle points under some realistic conditions. Our theoretical results are supported by detailed experiments.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 42 of its 69 references that are in the map. First posted on arXiv 19 Jan 2023.

Builds on

42 of 69 references.

References · 42

  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  3. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  4. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  5. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  6. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  7. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  8. 2017How to Escape Saddle Points Efficientlycited by 285
  9. 2017Three Factors Influencing Minima in SGDcited by 273
  10. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  11. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  12. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
30 more
  1. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  2. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  3. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  4. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  5. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  6. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  7. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  8. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  9. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  10. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  11. 2020Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamicscited by 96
  12. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 82
  13. 2022When Do Flat Minima Optimizers Work?cited by 78
  14. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  15. 2018Escaping Saddles with Stochastic Gradientscited by 66
  16. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  17. 2020Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Ratecited by 59
  18. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  19. 2019Loss Landscapes of Regularized Linear Autoencoderscited by 46
  20. 2016On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integratorscited by 35
  21. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  22. 2016The Power of Normalization: Faster Evasion of Saddle Pointscited by 29
  23. 2022Escaping Saddle Points for Effective Generalization on Class-Imbalanced Datacited by 28
  24. 2019Non-Gaussianity of Stochastic Gradient Noisecited by 22
  25. 2018Continuous-time Models for Stochastic Optimization Algorithmscited by 17
  26. 2023Stability Analysis of Sharpness-Aware Minimizationcited by 14
  27. 2020A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minimacited by 11
  28. 2022Rethinking Sharpness-Aware Minimization as Variational Inferencecited by 8
  29. 2022On the Theoretical Properties of Noise Correlation in Stochastic Optimizationcited by 5
  30. 2018Differential Equations for Modeling Asynchronous Algorithmscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 4; 2025: 7; 2026: 6.

22 papers.

Sort:

  1. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  2. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  3. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  4. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  5. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  6. 2026A Defense of the Quadratic Modelcited by 14
  7. 2023How to escape sharp minima with random perturbationscited by 13
  8. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  9. 2024SDEs for Minimax Optimizationcited by 6
  10. 2023On Statistical Properties of Sharpness-Aware Minimization: Provable Guaranteescited by 5
  11. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  12. 2025On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approachcited by 4
10 more
  1. 2023Critical Influence of Overparameterization on Sharpness-aware Minimizationcited by 3
  2. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  3. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  4. 2026Adaptive Methods Are Preferable in High Privacy Settings: An SDE Perspectivecited by 2
  5. 2025Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Modelscited by 0
  6. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  7. 2026X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correctioncited by 0
  8. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0
  9. 2026A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noisecited by 0
  10. 2026On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Boundscited by 0