mlsanityData to 14 Oct 2026

Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training

arXiv 2410.10373 ↗First posted 14 Oct 2024cs.LG stat.MLcited by 24

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) has substantially improved the generalization of neural networks under various settings. Despite the success, its effectiveness remains poorly understood. In this work, we discover an intriguing phenomenon in the training dynamics of SAM, shedding light on understanding its implicit bias towards flatter minima over Stochastic Gradient Descent (SGD). Specifically, we find that SAM efficiently selects flatter minima late in training. Remarkably, even a few epochs of SAM applied at the end of training yield nearly the same generalization and solution sharpness as full SAM training. Subsequently, we delve deeper into the underlying mechanism behind this phenomenon. Theoretically, we identify two phases in the learning dynamics after applying SAM late in training: i) SAM first escapes the minimum found by SGD exponentially fast; and ii) then rapidly converges to a flatter minimum within the same valley. Furthermore, we empirically investigate the role of SAM during the early training phase. We conjecture that the optimization method chosen in the late phase is more crucial in shaping the final solution's properties. Based on this viewpoint, we extend our findings from SAM to Adversarial Training.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 51 of its 65 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

51 of 65 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2016Wide Residual Networkscited by 3,070
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  7. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  10. 2017Exploring Generalization in Deep Learningcited by 600
  11. 2014Convex Optimization: Algorithms and Complexitycited by 575
  12. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
38 more
  1. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  2. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  3. 2017Three Factors Influencing Minima in SGDcited by 273
  4. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  5. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  6. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  7. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  8. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  9. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  10. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  11. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  12. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  13. 2020The Early Phase of Neural Network Trainingcited by 115
  14. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  15. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  16. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  17. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  18. 2022When Do Flat Minima Optimizers Work?cited by 78
  19. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  20. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  21. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  22. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  23. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
  24. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  25. 2020Noise and Fluctuation of Finite Learning Rate Stochastic Gradient Descentcited by 37
  26. 2021Power-law escape rate of SGDcited by 29
  27. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  28. 2021Stochastic gradient descent with noise of machine learning type. Part II: Continuous time analysiscited by 25
  29. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  30. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  31. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  32. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  33. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 17
  34. 2023Stability Analysis of Sharpness-Aware Minimizationcited by 14
  35. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 14
  36. 2023How to escape sharp minima with random perturbationscited by 13
  37. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  38. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 16.

24 papers.

Sort:

By area:

  1. 2026Sharpness-Aware Pretraining Mitigates Catastrophic Forgettingcited by 10
  2. 2026Too Sharp, Too Sure: When Calibration Follows Curvaturecited by 3
  3. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  4. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  5. 2025A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Biascited by 2
  6. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1
  7. 2025Do Flat Minima Improve Sparse Novel View Synthesis?cited by 1
  8. 2026Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuningcited by 1
  9. 2026A Faster Path to Continual Learningcited by 1
  10. 2026Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimizationcited by 1
  11. 2026When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Studycited by 1
  12. 20265% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuningcited by 1
12 more
  1. 2026Flat-Consensus Diffusion for Robust Data Reshaping under Noisy Evaluatorcited by 1
  2. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  3. 2025Convergence of optimizers implies eigenvalues filtering at equilibriumcited by 0
  4. 2025On the Tension Between Optimality and Adversarial Robustness in Policy Optimizationcited by 0
  5. 2026Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementationcited by 0
  6. 2026On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGDcited by 0
  7. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0
  8. 2026On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Boundscited by 0
  9. 2026PAPT++: Risk-Aware Adversarial Tuning and Generation for Single Domain Generalizationcited by 0
  10. 2026Temperon: Full-Time SAM Quality at a Third Less Wall-Clockcited by 0
  11. 2026Generalization Dynamics of LM Pre-trainingcited by 0
  12. 2026How Does Local Landscape Geometry Evolve in Language Model Pre-Training?cited by 0