mlsanityData to 14 Oct 2026

A Precise Characterization of SGD Stability Using Loss Surface Geometry

arXiv 2401.12332 ↗First posted 22 Jan 2024cs.LG math.OCcited by 2

Not on the map. Why?

Abstract (arXiv)

Stochastic Gradient Descent (SGD) stands as a cornerstone optimization algorithm with proven real-world empirical successes but relatively limited theoretical understanding. Recent research has illuminated a key factor contributing to its practical efficacy: the implicit regularization it instigates. Several studies have investigated the linear stability property of SGD in the vicinity of a stationary point as a predictive proxy for sharpness and generalization error in overparameterized neural networks (Wu et al., 2022; Jastrzebski et al., 2019; Cohen et al., 2021). In this paper, we delve deeper into the relationship between linear stability and sharpness. More specifically, we meticulously delineate the necessary and sufficient conditions for linear stability, contingent on hyperparameters of SGD and the sharpness at the optimum. Towards this end, we introduce a novel coherence measure of the loss Hessian that encapsulates pertinent geometric properties of the loss function that are relevant to the linear stability of SGD. It enables us to provide a simplified sufficient condition for identifying linear instability at an optimum. Notably, compared to previous works, our analysis relies on significantly milder assumptions and is applicable for a broader class of loss functions than known before, encompassing not only mean-squared error but also cross-entropy loss.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

29 of 37 references.

References · 29

  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  8. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  9. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  10. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  11. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  12. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
17 more
  1. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  2. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  3. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  4. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  5. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  6. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  7. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  8. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  9. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  10. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  11. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  12. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  13. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  14. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 16
  15. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  16. 2022Rethinking Sharpness-Aware Minimization as Variational Inferencecited by 8
  17. 2023Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descentcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Biascited by 2
  2. 2026Why Some Models Resist Unlearning: A Linear Stability Perspectivecited by 0