mlsanityData to 14 Oct 2026

Why Does Sharpness-Aware Minimization Generalize Better Than SGD?

arXiv 2310.07269 ↗First posted 11 Oct 2023cs.LG math.OC stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

The challenge of overfitting, in which the model memorizes the training data and fails to generalize to test data, has become increasingly significant in the training of large neural networks. To tackle this challenge, Sharpness-Aware Minimization (SAM) has emerged as a promising training method, which can improve the generalization of neural networks even in the presence of label noise. However, a deep understanding of how SAM works, especially in the setting of nonlinear neural networks and classification tasks, remains largely missing. This paper fills this gap by demonstrating why SAM generalizes better than Stochastic Gradient Descent (SGD) for a certain data model and two-layer convolutional ReLU networks. The loss landscape of our studied problem is nonsmooth, thus current explanations for the success of SAM based on the Hessian information are insufficient. Our result explains the benefits of SAM, particularly its ability to prevent noise learning in the early stages, thereby facilitating more effective learning of features. Experiments on both synthetic and real data corroborate our theory.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

28 of 37 references.

References · 28

  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2019Benign Overfitting in Linear Regressioncited by 549
  3. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  4. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  5. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  6. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  7. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  8. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  9. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  10. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  11. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  12. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
16 more
  1. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  2. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  3. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  4. 2020Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regimecited by 81
  5. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  6. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  7. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 68
  8. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  9. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  10. 2022Learning threshold neurons via the "edge of stability"cited by 51
  11. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  12. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 30
  13. 2022Data Augmentation as Feature Manipulationcited by 18
  14. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  15. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  16. 2022Improved Deep Neural Network Generalization Using m-Sharpness-Aware Minimizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 9; 2026: 6.

28 papers.

Sort:

By area:

  1. 2024Friendly Sharpness-Aware Minimizationcited by 53
  2. 2024Why is SAM Robust to Label Noise?cited by 21
  3. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 10
  4. 2024SURE: SUrvey REcipes for building reliable and robust deep networkscited by 8
  5. 2025Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Modelscited by 7
  6. 2024Benign overfitting in leaky ReLU networks with moderate input dimensioncited by 6
  7. 2024Tilted Sharpness-Aware Minimizationcited by 6
  8. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5
  9. 2024Provable Benefit of Cutout and CutMix for Feature Learningcited by 4
  10. 2025On the Role of Label Noise in the Feature Learning Processcited by 4
  11. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  12. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 4
16 more
  1. 2024Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalizationcited by 3
  2. 2024Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalizationcited by 3
  3. 2026Barriers to Discrete Reasoning with Transformers: A Survey Across Depth, Exactness, and Bandwidthcited by 3
  4. 2025Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Modelscited by 2
  5. 2025A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Biascited by 2
  6. 2024Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Ratecited by 1
  7. 2025SAMO: A Lightweight Sharpness-Aware Approach for Multi-Task Optimization with Joint Global-Local Perturbationcited by 1
  8. 2026Easy to Learn, Yet Hard to Forget: Towards Robust Unlearning Under Biascited by 1
  9. 2024Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamicscited by 0
  10. 2024Understanding SAM's Robustness to Noisy Labels through Gradient Down-weightingcited by 0
  11. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  12. 2025SAMOSA: Sharpness Aware Minimization for Open Set Active learningcited by 0
  13. 2026Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMscited by 0
  14. 2026Sharpness-Aware Poisoning: Enhancing Transferability of Injective Attacks on Recommender Systemscited by 0
  15. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0
  16. 2026Optimal Algorithms for Nonsmooth Nonconvex Stochastic Optimization with Constraintscited by 0