# Towards Understanding Sharpness-Aware Minimization

arXiv: https://arxiv.org/abs/2206.06232 · HTML: https://mlsanity.com/paper/2206.06232/

## Key facts

- Cited by 146 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Builds on (most cited first, 30 of 32)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Condition](https://mlsanity.com/paper/1608.04636/) (2016)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [Uniform convergence may be unable to explain generalization in deep learning](https://mlsanity.com/paper/1902.04742/) (2019)
- [SGD: General Analysis and Improved Rates](https://mlsanity.com/paper/1901.09401/) (2019)
- [Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process](https://mlsanity.com/paper/1904.09080/) (2019)
- [Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity](https://mlsanity.com/paper/2106.09524/) (2021)
- [Asymmetric Valleys: Beyond Sharp and Flat Local Minima](https://mlsanity.com/paper/1902.00744/) (2019)
- [Information-Theoretic Generalization Bounds for Stochastic Gradient Descent](https://mlsanity.com/paper/2102.00931/) (2021)
- [Regularizing Neural Networks via Adversarial Model Perturbation](https://mlsanity.com/paper/2010.04925/) (2020)
- [Bad Global Minima Exist and SGD Can Reach Them](https://mlsanity.com/paper/1906.02613/) (2019)
- [On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications](https://mlsanity.com/paper/2110.03128/) (2021)
- [SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning](https://mlsanity.com/paper/1805.07898/) (2018)

## Built on by (most cited first, 30 of 146)

- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond](https://mlsanity.com/paper/2502.05374/) (2025)
- [How Does Sharpness-Aware Minimization Minimize Sharpness?](https://mlsanity.com/paper/2211.05729/) (2022)
- [On the Maximum Hessian Eigenvalue and Generalization](https://mlsanity.com/paper/2206.10654/) (2022)
- [Friendly Sharpness-Aware Minimization](https://mlsanity.com/paper/2403.12350/) (2024)
- [Deep Networks Always Grok and Here is Why](https://mlsanity.com/paper/2402.15555/) (2024)
- [FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracy](https://mlsanity.com/paper/2302.10429/) (2023)
- [Improving the Model Consistency of Decentralized Federated Learning](https://mlsanity.com/paper/2302.04083/) (2023)
- [An Adaptive Policy to Employ Sharpness-Aware Minimization](https://mlsanity.com/paper/2304.14647/) (2023)
- [Sharpness-Aware Minimization Leads to Low-Rank Features](https://mlsanity.com/paper/2305.16292/) (2023)
- [Enhancing Sharpness-Aware Optimization Through Variance Suppression](https://mlsanity.com/paper/2309.15639/) (2023)
- [Does SGD really happen in tiny subspaces?](https://mlsanity.com/paper/2405.16002/) (2024)
- [Make Continual Learning Stronger via C-Flat](https://mlsanity.com/paper/2404.00986/) (2024)
- [Make Landscape Flatter in Differentially Private Federated Learning](https://mlsanity.com/paper/2303.11242/) (2023)
- [Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability](https://mlsanity.com/paper/2307.07873/) (2023)
- [Why Does Sharpness-Aware Minimization Generalize Better Than SGD?](https://mlsanity.com/paper/2310.07269/) (2023)
- [Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization](https://mlsanity.com/paper/2405.18890/) (2024)
- [Normalization Layers Are All That Sharpness-Aware Minimization Needs](https://mlsanity.com/paper/2306.04226/) (2023)
- [On Efficient Training of Large-Scale Deep Learning Models: A Literature Review](https://mlsanity.com/paper/2304.03589/) (2023)
- [The Crucial Role of Normalization in Sharpness-Aware Minimization](https://mlsanity.com/paper/2305.15287/) (2023)
- [SAM operates far from home: eigenvalue regularization as a dynamical phenomenon](https://mlsanity.com/paper/2302.08692/) (2023)
- [On the Duality Between Sharpness-Aware Minimization and Adversarial Training](https://mlsanity.com/paper/2402.15152/) (2024)
- [Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training](https://mlsanity.com/paper/2410.10373/) (2024)
- [An SDE for Modeling SAM: Theory and Insights](https://mlsanity.com/paper/2301.08203/) (2023)
- [AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks](https://mlsanity.com/paper/2303.00565/) (2023)
- [Why is SAM Robust to Label Noise?](https://mlsanity.com/paper/2405.03676/) (2024)
- [Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization](https://mlsanity.com/paper/2501.12666/) (2025)
- [Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima](https://mlsanity.com/paper/2306.09850/) (2023)
- [FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learning](https://mlsanity.com/paper/2310.16412/) (2023)
- [Sharpness-Aware Minimization and the Edge of Stability](https://mlsanity.com/paper/2309.12488/) (2023)

## Abstract (arXiv)

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based on a PAC-Bayes generalization bound and the idea of convergence to flat minima are incomplete. Moreover, there are no explanations for the success of using m-sharpness in SAM which has been shown as essential for generalization. To better understand this aspect of SAM, we theoretically analyze its implicit bias for diagonal linear networks. We prove that SAM always chooses a solution that enjoys better generalization properties than standard gradient descent for a certain class of problems, and this effect is amplified by using m-sharpness. We further study the properties of the implicit bias on non-linear networks empirically, where we show that fine-tuning a standard model with SAM can lead to significant generalization improvements. Finally, we provide convergence results of SAM for non-convex objectives when used with stochastic gradients. We illustrate these results empirically for deep networks and discuss their relation to the generalization behavior of SAM. The code of our experiments is available at https://github.com/tml-epfl/understanding-sam.