mlsanityData to 7 Oct 2026

SAM operates far from home: eigenvalue regularization as a dynamical phenomenon

arXiv 2302.08692 ↗First posted 17 Feb 2023cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

The Sharpness Aware Minimization (SAM) optimization algorithm has been shown to control large eigenvalues of the loss Hessian and provide generalization benefits in a variety of settings. The original motivation for SAM was a modified loss function which penalized sharp minima; subsequent analyses have also focused on the behavior near minima. However, our work reveals that SAM provides a strong regularization of the eigenvalues throughout the learning trajectory. We show that in a simplified setting, SAM dynamically induces a stabilization related to the edge of stability (EOS) phenomenon observed in large learning rate gradient descent. Our theory predicts the largest eigenvalue as a function of the learning rate and SAM radius parameters. Finally, we show that practical models can also exhibit this EOS stabilization, and that understanding SAM must account for these dynamics far away from any minima.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

26 of 37 references.

References · 26

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2016Wide Residual Networkscited by 3,057
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  10. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  11. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  12. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
14 more
  1. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  2. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  3. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  4. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  5. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  6. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  7. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  8. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  9. 2022Second-order regression models exhibit progressive sharpening to the edge of stabilitycited by 40
  10. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  11. 2020Temperature check: theory and practice for training models with softmax-cross-entropy lossescited by 31
  12. 2022Homogenization of SGD in high-dimensions: Exact dynamics and generalization propertiescited by 26
  13. 2022Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensionscited by 23
  14. 2022Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensionscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 7; 2025: 6; 2026: 3.

24 papers.

Sort:

By area:

  1. 2024Does SGD really happen in tiny subspaces?cited by 35
  2. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  3. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  4. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  5. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  6. 2023Sharpness-Aware Minimization and the Edge of Stabilitycited by 16
  7. 2023From Stability to Chaos: Analyzing Gradient Descent Dynamics in Quadratic Regressioncited by 16
  8. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  9. 2024High dimensional analysis reveals conservative sharpening and a stochastic edge of stabilitycited by 15
  10. 2023Noise Stability Optimization for Finding Flat Minima: A Hessian-based Regularization Approachcited by 14
  11. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  12. 2024Neglected Hessian component explains mysteries in Sharpness regularizationcited by 12
12 more
  1. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  2. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  3. 2023On Statistical Properties of Sharpness-Aware Minimization: Provable Guaranteescited by 5
  4. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 3
  5. 2024A Precise Characterization of SGD Stability Using Loss Surface Geometrycited by 2
  6. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 2
  7. 2025Variational Learning Finds Flatter Solutions at the Edge of Stabilitycited by 2
  8. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  9. 2025Convergence of optimizers implies eigenvalues filtering at equilibriumcited by 0
  10. 2026Why Some Models Resist Unlearning: A Linear Stability Perspectivecited by 0
  11. 2026Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localizationcited by 0
  12. 2026Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimizationcited by 0