mlsanityData to 7 Oct 2026

An Adaptive Policy to Employ Sharpness-Aware Minimization

arXiv 2304.14647 ↗First posted 28 Apr 2023cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, since each SAM update requires computing two gradients, its computational cost and training time are both doubled compared to standard empirical risk minimization (ERM). Recent state-of-the-arts reduce the fraction of SAM updates and thus accelerate SAM by switching between SAM and ERM updates randomly or periodically. In this paper, we design an adaptive policy to employ SAM based on the loss landscape geometry. Two efficient algorithms, AE-SAM and AE-LookSAM, are proposed. We theoretically show that AE-SAM has the same convergence rate as SAM. Experimental results on various datasets and architectures demonstrate the efficiency and effectiveness of the adaptive policy.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 28 Apr 2023.

Builds on

33 of 43 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016Wide Residual Networkscited by 3,057
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
21 more
  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  2. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  3. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  4. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  5. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  6. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  7. 2016Deep Pyramidal Residual Networkscited by 163
  8. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  9. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  10. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
  11. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  12. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  13. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  14. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  15. 2022Sharpness-Aware Training for Freecited by 95
  16. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  17. 2020Relative Flatness and Generalizationcited by 88
  18. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  19. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  20. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  21. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 19; 2025: 14; 2026: 6.

43 papers.

Sort:

By area:

  1. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  2. 2024Friendly Sharpness-Aware Minimizationcited by 52
  3. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  4. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  5. 2023Adaptive Sharpness-Aware Pruning for Robust Sparse Networkscited by 17
  6. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  7. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  8. 2024Domain-Inspired Sharpness-Aware Minimization Under Domain Shiftscited by 12
  9. 2023CR-SAM: Curvature Regularized Sharpness-Aware Minimizationcited by 10
  10. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
  11. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  12. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
31 more
  1. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  2. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  3. 2024Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Modelscited by 7
  4. 2025Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Modelscited by 7
  5. 2024Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimizationcited by 5
  6. 2024Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classificationcited by 4
  7. 2025Towards Robust Influence Functions with Flat Validation Minimacited by 4
  8. 2024Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategycited by 3
  9. 2024Sharpness-diversity tradeoff: improving flat ensembles with SharpBalancecited by 3
  10. 2025Learning from Loss Landscape: Generalizable Mixed-Precision Quantization via Adaptive Sharpness-Aware Gradient Aligningcited by 3
  11. 2025Curvature Learning for Generalization of Hyperbolic Neural Networkscited by 3
  12. 2025Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Modelscited by 3
  13. 2024Bilateral Sharpness-Aware Minimization for Flatter Minimacited by 2
  14. 2024Sharpness-Aware Black-Box Optimizationcited by 2
  15. 2025Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Searchcited by 2
  16. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  17. 2024Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimizationcited by 1
  18. 2024Enhancing Sharpness-Aware Minimization by Learning Perturbation Radiuscited by 1
  19. 2025Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithmcited by 1
  20. 2025Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learningcited by 1
  21. 2025VASSO: Variance Suppression for Sharpness-Aware Minimizationcited by 1
  22. 2026A Faster Path to Continual Learningcited by 1
  23. 2024SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Datacited by 0
  24. 2024Understanding SAM's Robustness to Noisy Labels through Gradient Down-weightingcited by 0
  25. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  26. 2025Adaptively Sampling-Reusing-Mixing Decomposed Gradients to Speed Up Sharpness Aware Minimizationcited by 0
  27. 2026Incorruptible Neural Networks: Training Models that can Generalize to Large Internal Perturbationscited by 0
  28. 2026Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizerscited by 0
  29. 2026Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementationcited by 0
  30. 2026Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimizationcited by 0
  31. 2026EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbationscited by 0