# Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training

arXiv: https://arxiv.org/abs/2410.10373 · HTML: https://mlsanity.com/paper/2410.10373/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 65 references that are in the map.
- First posted on arXiv 14 Oct 2024.

## Builds on (most cited first, 30 of 51)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Convex Optimization: Algorithms and Complexity](https://mlsanity.com/paper/1405.4980/) (2014)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Three Factors Influencing Minima in SGD](https://mlsanity.com/paper/1711.04623/) (2017)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Surrogate Gap Minimization Improves Sharpness-Aware Training](https://mlsanity.com/paper/2203.08065/) (2022)
- [The Break-Even Point on Optimization Trajectories of Deep Neural Networks](https://mlsanity.com/paper/2002.09572/) (2020)
- [Towards Understanding Sharpness-Aware Minimization](https://mlsanity.com/paper/2206.06232/) (2022)
- [Efficient Sharpness-aware Minimization for Improved Training of Neural Networks](https://mlsanity.com/paper/2110.03141/) (2021)
- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)
- [Understanding Gradient Descent on Edge of Stability in Deep Learning](https://mlsanity.com/paper/2205.09745/) (2022)
- [Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process](https://mlsanity.com/paper/1904.09080/) (2019)
- [Label Noise SGD Provably Prefers Flat Global Minimizers](https://mlsanity.com/paper/2106.06530/) (2021)
- [The Early Phase of Neural Network Training](https://mlsanity.com/paper/2002.10365/) (2020)
- [Sharpness-Aware Minimization Improves Language Model Generalization](https://mlsanity.com/paper/2110.08529/) (2021)
- [What Happens after SGD Reaches Zero Loss? --A Mathematical Framework](https://mlsanity.com/paper/2110.06914/) (2021)
- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [On the Implicit Bias in Deep-Learning Algorithms](https://mlsanity.com/paper/2208.12591/) (2022)
- [When Do Flat Minima Optimizers Work?](https://mlsanity.com/paper/2202.00661/) (2022)

## Built on by (most cited first, 24 of 24)

- [Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting](https://mlsanity.com/paper/2605.02105/) (2026)
- [Too Sharp, Too Sure: When Calibration Follows Curvature](https://mlsanity.com/paper/2604.20614/) (2026)
- [Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization](https://mlsanity.com/paper/2505.23866/) (2025)
- [Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise](https://mlsanity.com/paper/2509.18001/) (2025)
- [A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias](https://mlsanity.com/paper/2511.17378/) (2025)
- [R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization](https://mlsanity.com/paper/2510.07758/) (2025)
- [Do Flat Minima Improve Sparse Novel View Synthesis?](https://mlsanity.com/paper/2511.17918/) (2025)
- [Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning](https://mlsanity.com/paper/2602.09395/) (2026)
- [A Faster Path to Continual Learning](https://mlsanity.com/paper/2604.11064/) (2026)
- [Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization](https://mlsanity.com/paper/2605.10183/) (2026)
- [When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study](https://mlsanity.com/paper/2605.26929/) (2026)
- [5% > 100%: Flatness Preference is All You Need for Multimodal Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2606.10488/) (2026)
- [Flat-Consensus Diffusion for Robust Data Reshaping under Noisy Evaluator](https://mlsanity.com/paper/2609.32696/) (2026)
- [Sharpness-Aware Minimization Can Hallucinate Minimizers](https://mlsanity.com/paper/2509.21818/) (2025)
- [Convergence of optimizers implies eigenvalues filtering at equilibrium](https://mlsanity.com/paper/2510.09034/) (2025)
- [On the Tension Between Optimality and Adversarial Robustness in Policy Optimization](https://mlsanity.com/paper/2512.01228/) (2025)
- [Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation](https://mlsanity.com/paper/2603.10048/) (2026)
- [On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD](https://mlsanity.com/paper/2603.10397/) (2026)
- [Certification of Machine Learning Models via Directional Sharpness](https://mlsanity.com/paper/2606.25004/) (2026)
- [On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds](https://mlsanity.com/paper/2608.03197/) (2026)
- [PAPT++: Risk-Aware Adversarial Tuning and Generation for Single Domain Generalization](https://mlsanity.com/paper/2609.04837/) (2026)
- [Temperon: Full-Time SAM Quality at a Third Less Wall-Clock](https://mlsanity.com/paper/2609.17575/) (2026)
- [Generalization Dynamics of LM Pre-training](https://mlsanity.com/paper/2609.33150/) (2026)
- [How Does Local Landscape Geometry Evolve in Language Model Pre-Training?](https://mlsanity.com/paper/2609.39767/) (2026)

## Abstract (arXiv)

Sharpness-Aware Minimization (SAM) has substantially improved the generalization of neural networks under various settings. Despite the success, its effectiveness remains poorly understood. In this work, we discover an intriguing phenomenon in the training dynamics of SAM, shedding light on understanding its implicit bias towards flatter minima over Stochastic Gradient Descent (SGD). Specifically, we find that SAM efficiently selects flatter minima late in training. Remarkably, even a few epochs of SAM applied at the end of training yield nearly the same generalization and solution sharpness as full SAM training. Subsequently, we delve deeper into the underlying mechanism behind this phenomenon. Theoretically, we identify two phases in the learning dynamics after applying SAM late in training: i) SAM first escapes the minimum found by SGD exponentially fast; and ii) then rapidly converges to a flatter minimum within the same valley. Furthermore, we empirically investigate the role of SAM during the early training phase. We conjecture that the optimization method chosen in the late phase is more crucial in shaping the final solution's properties. Based on this viewpoint, we extend our findings from SAM to Adversarial Training.