mlsanityData to 14 Oct 2026

Exploring the Effect of Multi-step Ascent in Sharpness-Aware Minimization

arXiv 2302.10181 ↗First posted 27 Jan 2023cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Recently, Sharpness-Aware Minimization (SAM) has shown state-of-the-art performance by seeking flat minima. To minimize the maximum loss within a neighborhood in the parameter space, SAM uses an ascent step, which perturbs the weights along the direction of gradient ascent with a given radius. While single-step or multi-step can be taken during ascent steps, previous studies have shown that multi-step ascent SAM rarely improves generalization performance. However, this phenomenon is particularly interesting because the multi-step ascent is expected to provide a better approximation of the maximum neighborhood loss. Therefore, in this paper, we analyze the effect of the number of ascent steps and investigate the difference between both single-step ascent SAM and multi-step ascent SAM. We identify the effect of the number of ascent on SAM optimization and reveal that single-step ascent SAM and multi-step ascent SAM exhibit distinct loss landscapes. Based on these observations, we finally suggest a simple modification that can mitigate the inefficiency of multi-step ascent SAM.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

28 of 42 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2016Wide Residual Networkscited by 3,070
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  11. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  12. 2017Exploring Generalization in Deep Learningcited by 600
16 more
  1. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  3. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  4. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  5. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 292
  6. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
  7. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  8. 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 158
  9. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  10. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  11. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  12. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  13. 2020Understanding Catastrophic Overfitting in Single-step Adversarial Trainingcited by 76
  14. 2017Gradient Descent Can Take Exponential Time to Escape Saddle Pointscited by 74
  15. 2020Do We Need Zero Training Loss After Achieving Zero Training Error?cited by 61
  16. 2023Stability Analysis of Sharpness-Aware Minimizationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 1; 2026: 1.

9 papers.

  1. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  2. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  3. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  4. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  5. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  6. 2023Lookbehind-SAM: k steps back, 1 step forwardcited by 5
  7. 2025VASSO: Variance Suppression for Sharpness-Aware Minimizationcited by 1
  8. 20241st-Order Magic: Analysis of Sharpness-Aware Minimizationcited by 0
  9. 2026Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementationcited by 0