mlsanityData to 7 Oct 2026

A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias

arXiv 2511.17378 ↗First posted 21 Nov 2025cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Understanding the dynamics of optimization in deep learning is increasingly important as models scale. While stochastic gradient descent (SGD) and its variants reliably find solutions that generalize well, the mechanisms driving this generalization remain unclear. Notably, these algorithms often prefer flatter or simpler minima, particularly in overparameterized settings. Prior work has linked flatness to generalization, and methods like Sharpness-Aware Minimization (SAM) explicitly encourage flatness, but a unified theory connecting data structure, optimization dynamics, and the nature of learned solutions is still lacking. In this work, we develop a linear stability framework that analyzes the behavior of SGD, random perturbations, and SAM, particularly in two layer ReLU networks. Central to our analysis is a coherence measure that quantifies how gradient curvature aligns across data points, revealing why certain minima are stable and favored during training.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 35 of its 40 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

35 of 40 references.

References · 35

  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  4. 2017A Closer Look at Memorization in Deep Networkscited by 936
  5. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  7. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  9. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  10. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  11. 2017Implicit Regularization in Matrix Factorizationcited by 318
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
23 more
  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  3. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  4. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  5. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  6. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  7. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  8. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  9. 2018Interpreting Black Box Predictions using Fisher Kernelscited by 48
  10. 2019The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descentcited by 46
  11. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  12. 2020Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimizationcited by 41
  13. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  14. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  15. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  16. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  17. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  18. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  19. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  20. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  21. 2023Exact Mean Square Linear Stability Analysis for SGDcited by 10
  22. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  23. 2024A Precise Characterization of SGD Stability Using Loss Surface Geometrycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Why Some Models Resist Unlearning: A Linear Stability Perspectivecited by 0
  2. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0