mlsanityData to 14 Oct 2026

Momentum-SAM: Sharpness Aware Minimization without Computational Overhead

arXiv 2401.12033 ↗First posted 22 Jan 2024cs.LG cs.CVcited by 14

Not on the map. Why?

Abstract (arXiv)

The recently proposed optimization algorithm for deep neural networks Sharpness Aware Minimization (SAM) suggests perturbing parameters before gradient calculation by a gradient ascent step to guide the optimization into parameter space regions of flat loss. While significant generalization improvements and thus reduction of overfitting could be demonstrated, the computational costs are doubled due to the additionally needed gradient calculation, making SAM unfeasible in case of limited computationally capacities. Motivated by Nesterov Accelerated Gradient (NAG) we propose Momentum-SAM (MSAM), which perturbs parameters in the direction of the accumulated momentum vector to achieve low sharpness without significant computational overhead or memory demands over SGD or Adam. We evaluate MSAM in detail and reveal insights on separable mechanisms of NAG, SAM and MSAM regarding training optimization and generalization. Code is available at https://github.com/MarlonBecker/MSAM.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 22 Jan 2024.

Builds on

42 of 59 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2014Going Deeper with Convolutionscited by 4,241
  9. 2016Wide Residual Networkscited by 3,070
  10. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
30 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  3. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  4. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  5. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  6. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  7. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  8. 2017Training GANs with Optimismcited by 217
  9. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  10. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  11. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  12. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  13. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  14. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  15. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  16. 2022Better plain ViT baselines for ImageNet-1kcited by 98
  17. 2022Sharpness-Aware Training for Freecited by 95
  18. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  19. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  20. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  21. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  22. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  23. 2020On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Themcited by 59
  24. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  25. 2022SAM as an Optimal Relaxation of Bayescited by 41
  26. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  27. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  28. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  29. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  30. 2020Extrapolation for Large-batch Training in Deep Learningcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 4.

14 papers.

Sort:

  1. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  2. 2025Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Schedulercited by 12
  3. 2025LoRA-MGPO: Mitigating Double Descent in Low-Rank Adaptation via Momentum-Guided Perturbation Optimizationcited by 10
  4. 2024Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimizationcited by 5
  5. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 3
  6. 2026Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluationscited by 3
  7. 2024Bilateral Sharpness-Aware Minimization for Flatter Minimacited by 2
  8. 2025FairSAM: Fair Classification on Corrupted Image Data Through Sharpness-Aware Minimizationcited by 1
  9. 2025CurvaDion: Curvature-Adaptive Distributed Orthonormalizationcited by 1
  10. 2026Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimizationcited by 1
  11. 2026Sharpness-Aware Minimization and Muon: Robustness under the Spectral Normcited by 1
  12. 2024Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awarenesscited by 0
2 more
  1. 2025Sharpness-aware Federated Graph Learningcited by 0
  2. 2026EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbationscited by 0