mlsanityData to 14 Oct 2026

Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise

arXiv 2509.18001 ↗First posted 22 Sep 2025cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware minimization (SAM) has emerged as a highly effective technique to improve model generalization, but its underlying principles are not fully understood. We investigate m-sharpness, where SAM performance improves monotonically as the micro-batch size for computing perturbations decreases, a phenomenon critical for distributed training yet lacking rigorous explanation. We leverage an extended Stochastic Differential Equation (SDE) framework and analyze stochastic gradient noise (SGN) to characterize the dynamics of SAM variants, including n-SAM and m-SAM. Our analysis reveals that stochastic perturbations induce an implicit variance-based sharpness regularization whose strength increases as m decreases. Motivated by this insight, we propose Reweighted SAM (RW-SAM), which employs sharpness-weighted sampling to mimic the generalization benefits of m-SAM while remaining parallelizable. Comprehensive experiments validate our theory and method.Code is available at https://github.com/RitianLuo/RW-SAM.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 70 of its 88 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

70 of 88 references.

References · 70

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2016Wide Residual Networkscited by 3,070
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  10. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  11. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  12. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
38 more
  1. 2017Three Factors Influencing Minima in SGDcited by 273
  2. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  3. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  4. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  5. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  6. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  7. 2015Stochastic modified equations and adaptive stochastic gradient algorithmscited by 142
  8. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  9. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  10. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  11. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  12. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  13. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  14. 2022Sharpness-Aware Training for Freecited by 95
  15. 2022Vision Transformers provably learn spatial structurecited by 93
  16. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  17. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  18. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  19. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  20. 2022When Do Flat Minima Optimizers Work?cited by 78
  21. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  22. 2018Escaping Saddles with Stochastic Gradientscited by 66
  23. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  24. 2024Friendly Sharpness-Aware Minimizationcited by 53
  25. 2019Bad Global Minima Exist and SGD Can Reach Themcited by 49
  26. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  27. 2022SAM as an Optimal Relaxation of Bayescited by 41
  28. 2021Strength of Minibatch Noise in SGDcited by 39
  29. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  30. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  31. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  32. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  33. 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenoncited by 24
  34. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  35. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  36. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  37. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  38. 2024On the Feature Learning in Diffusion Modelscited by 20

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  2. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1