mlsanityData to 7 Oct 2026

Sharpness-Aware Minimization and the Edge of Stability

arXiv 2309.12488 ↗First posted 21 Sep 2023cs.LG cs.NE stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

Recent experiments have shown that, often, when training a neural network with gradient descent (GD) with a step size η, the operator norm of the Hessian of the loss grows until it approximately reaches 2/η, after which it fluctuates around this value. The quantity 2/η has been called the "edge of stability" based on consideration of a local quadratic approximation of the loss. We perform a similar calculation to arrive at an "edge of stability" for Sharpness-Aware Minimization (SAM), a variant of GD which has been shown to improve its generalization. Unlike the case for GD, the resulting SAM-edge depends on the norm of the gradient. Using three deep learning training tasks, we see empirically that SAM operates on the edge of stability identified by this analysis.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 17 of its 23 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

17 of 23 references.

References · 17

  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  2. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  3. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  4. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  5. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  6. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  7. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  8. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  9. 2022Understanding the unstable convergence of gradient descentcited by 84
  10. 2022Understanding Edge-of-Stability Training Dynamics with a Minimalist Examplecited by 54
  11. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  12. 2022Learning threshold neurons via the "edge of stability"cited by 50
5 more
  1. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 42
  2. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  3. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  4. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  5. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 4; 2026: 4.

16 papers.

Sort:

  1. 2024Does SGD really happen in tiny subspaces?cited by 35
  2. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  3. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  4. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  5. 2026Non-Euclidean Gradient Descent Operates at the Edge of Stabilitycited by 7
  6. 2024Tilted Sharpness-Aware Minimizationcited by 6
  7. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  8. 2024Towards certifiable AI in aviation: landscape, challenges, and opportunitiescited by 2
  9. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  10. 2025Spectral Alignment as Predictor of Loss Explosion in Neural Network Trainingcited by 2
  11. 2024Training on the Edge of Stability Is Caused by Layerwise Jacobian Alignmentcited by 1
  12. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
4 more
  1. 2025Zeroth-Order Sharpness-Aware Learning with Exponential Tiltingcited by 0
  2. 2026Zeroth-Order Optimization at the Edge of Stabilitycited by 0
  3. 2026Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimizationcited by 0
  4. 2026On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Boundscited by 0