# An Adaptive Policy to Employ Sharpness-Aware Minimization

arXiv: https://arxiv.org/abs/2304.14647 · HTML: https://mlsanity.com/paper/2304.14647/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 28 Apr 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Surrogate Gap Minimization Improves Sharpness-Aware Training](https://mlsanity.com/paper/2203.08065/) (2022)
- [Deep Pyramidal Residual Networks](https://mlsanity.com/paper/1610.02915/) (2016)
- [Towards Understanding Sharpness-Aware Minimization](https://mlsanity.com/paper/2206.06232/) (2022)
- [Stochastic Variance Reduction for Nonconvex Optimization](https://mlsanity.com/paper/1603.06160/) (2016)
- [Generalized Federated Learning via Sharpness Aware Minimization](https://mlsanity.com/paper/2206.02618/) (2022)
- [Efficient Sharpness-aware Minimization for Improved Training of Neural Networks](https://mlsanity.com/paper/2110.03141/) (2021)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [Towards Efficient and Scalable Sharpness-Aware Minimization](https://mlsanity.com/paper/2203.02714/) (2022)
- [Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning](https://mlsanity.com/paper/2202.03599/) (2022)
- [Sharpness-Aware Training for Free](https://mlsanity.com/paper/2205.14083/) (2022)
- [Asymmetric Valleys: Beyond Sharp and Flat Local Minima](https://mlsanity.com/paper/1902.00744/) (2019)
- [Relative Flatness and Generalization](https://mlsanity.com/paper/2001.00939/) (2020)
- [Fisher SAM: Information Geometry and Sharpness Aware Minimisation](https://mlsanity.com/paper/2206.04920/) (2022)

## Built on by (most cited first, 30 of 43)

- [Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond](https://mlsanity.com/paper/2502.05374/) (2025)
- [Friendly Sharpness-Aware Minimization](https://mlsanity.com/paper/2403.12350/) (2024)
- [Enhancing Sharpness-Aware Optimization Through Variance Suppression](https://mlsanity.com/paper/2309.15639/) (2023)
- [Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima](https://mlsanity.com/paper/2306.09850/) (2023)
- [Adaptive Sharpness-Aware Pruning for Robust Sparse Networks](https://mlsanity.com/paper/2306.14306/) (2023)
- [Momentum-SAM: Sharpness Aware Minimization without Computational Overhead](https://mlsanity.com/paper/2401.12033/) (2024)
- [Revisiting Random Weight Perturbation for Efficiently Improving Generalization](https://mlsanity.com/paper/2404.00357/) (2024)
- [Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts](https://mlsanity.com/paper/2405.18861/) (2024)
- [CR-SAM: Curvature Regularized Sharpness-Aware Minimization](https://mlsanity.com/paper/2312.13555/) (2023)
- [Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems](https://mlsanity.com/paper/2410.14802/) (2024)
- [Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes](https://mlsanity.com/paper/2412.13573/) (2024)
- [Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy](https://mlsanity.com/paper/2401.07250/) (2024)
- [SAMPa: Sharpness-aware Minimization Parallelized](https://mlsanity.com/paper/2410.10683/) (2024)
- [MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generation](https://mlsanity.com/paper/2510.07835/) (2025)
- [Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models](https://mlsanity.com/paper/2406.13137/) (2024)
- [Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2510.08625/) (2025)
- [Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimization](https://mlsanity.com/paper/2406.08001/) (2024)
- [Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification](https://mlsanity.com/paper/2410.16845/) (2024)
- [Towards Robust Influence Functions with Flat Validation Minima](https://mlsanity.com/paper/2505.19097/) (2025)
- [Curvature Learning for Generalization of Hyperbolic Neural Networks](https://mlsanity.com/paper/2508.17232/) (2025)
- [Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy](https://mlsanity.com/paper/2407.03641/) (2024)
- [Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance](https://mlsanity.com/paper/2407.12996/) (2024)
- [Learning from Loss Landscape: Generalizable Mixed-Precision Quantization via Adaptive Sharpness-Aware Gradient Aligning](https://mlsanity.com/paper/2505.04877/) (2025)
- [Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models](https://mlsanity.com/paper/2508.19564/) (2025)
- [Bilateral Sharpness-Aware Minimization for Flatter Minima](https://mlsanity.com/paper/2409.13173/) (2024)
- [Sharpness-Aware Black-Box Optimization](https://mlsanity.com/paper/2410.12457/) (2024)
- [Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Search](https://mlsanity.com/paper/2503.10404/) (2025)
- [Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning](https://mlsanity.com/paper/2503.11147/) (2025)
- [Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization](https://mlsanity.com/paper/2505.23866/) (2025)
- [Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimization](https://mlsanity.com/paper/2403.08821/) (2024)

## Abstract (arXiv)

Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, since each SAM update requires computing two gradients, its computational cost and training time are both doubled compared to standard empirical risk minimization (ERM). Recent state-of-the-arts reduce the fraction of SAM updates and thus accelerate SAM by switching between SAM and ERM updates randomly or periodically. In this paper, we design an adaptive policy to employ SAM based on the loss landscape geometry. Two efficient algorithms, AE-SAM and AE-LookSAM, are proposed. We theoretically show that AE-SAM has the same convergence rate as SAM. Experimental results on various datasets and architectures demonstrate the efficiency and effectiveness of the adaptive policy.