mlsanityData to 14 Oct 2026

On Statistical Properties of Sharpness-Aware Minimization: Provable Guarantees

arXiv 2302.11836 ↗First posted 23 Feb 2023stat.ML cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) is a recent optimization framework aiming to improve the deep neural network generalization, through obtaining flatter (i.e. less sharp) solutions. As SAM has been numerically successful, recent papers have studied the theoretical aspects of the framework and have shown SAM solutions are indeed flat. However, there has been limited theoretical exploration regarding statistical properties of SAM. In this work, we directly study the statistical performance of SAM, and present a new theoretical explanation of why SAM generalizes well. To this end, we study two statistical problems, neural networks with a hidden layer and kernel regression, and prove under certain conditions, SAM has smaller prediction error over Gradient Descent (GD). Our results concern both convex and non-convex settings, and show that SAM is particularly well-suited for non-convex problems. Additionally, we prove that in our setup, SAM solutions are less sharp as well, showing our results are in agreement with the previous work. Our theoretical findings are validated using numerical experiments on numerous scenarios, including deep neural networks.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

30 of 42 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  6. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  7. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  8. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
  9. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  10. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  11. 2018To understand deep learning we need to understand kernel learningcited by 187
  12. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
18 more
  1. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  2. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  3. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  4. 2018Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysiscited by 110
  5. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  6. 2022Sharpness-Aware Training for Freecited by 95
  7. 2022When Do Flat Minima Optimizers Work?cited by 78
  8. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  9. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  10. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  11. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  12. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  13. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  14. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  15. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  16. 2022Rethinking Sharpness-Aware Minimization as Variational Inferencecited by 8
  17. 2022Improved Deep Neural Network Generalization Using m-Sharpness-Aware Minimizationcited by 5
  18. 2021Minimum sharpness: Scale-invariant parameter-robustness of neural networkscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 5.

5 papers.

  1. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  2. 2024Why is SAM Robust to Label Noise?cited by 21
  3. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  4. 2024Sharpness-diversity tradeoff: improving flat ensembles with SharpBalancecited by 3
  5. 2024Understanding SAM's Robustness to Noisy Labels through Gradient Down-weightingcited by 0