mlsanityData to 7 Oct 2026

ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks

arXiv 2102.11600 ↗First posted 23 Feb 2021cs.LG stat.MLcited by 293

Not on the map. Why?

Abstract (arXiv)

Recently, learning algorithms motivated from sharpness of loss surface as an effective measure of generalization gap have shown state-of-the-art performances. Nevertheless, sharpness defined in a rigid region with a fixed radius, has a drawback in sensitivity to parameter re-scaling which leaves the loss unaffected, leading to weakening of the connection between sharpness and generalization gap. In this paper, we introduce the concept of adaptive sharpness which is scale-invariant and propose the corresponding generalization bound. We suggest a novel learning method, adaptive sharpness-aware minimization (ASAM), utilizing the proposed generalization bound. Experimental results in various benchmark datasets show that ASAM contributes to significant improvement of model generalization performance.

Key facts. Cited by 293 papers in the map (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 23 Feb 2021.

Builds on

28 of 41 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016Densely Connected Convolutional Networkscited by 6,691
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2016Wide Residual Networkscited by 3,057
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
16 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2019When Does Label Smoothing Help?cited by 985
  3. 2017Exploring Generalization in Deep Learningcited by 595
  4. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  5. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  6. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  7. 2016Deep Pyramidal Residual Networkscited by 163
  8. 2015Learning with Symmetric Label Noise: The Importance of Being Unhingedcited by 120
  9. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 99
  10. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 58
  11. 2018ShakeDrop Regularization for Deep Residual Learningcited by 40
  12. 2019The intriguing role of module criticality in the generalization of deep networkscited by 40
  13. 2020Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruptioncited by 26
  14. 2019The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networkscited by 20
  15. 2019Positively Scale-Invariant Flatness of ReLU Neural Networkscited by 11
  16. 2016Training Recurrent Neural Networks by Diffusioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 43; 2023: 64; 2024: 72; 2025: 54; 2026: 49.

293 papers.

Sort:

By area:

  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  3. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  4. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  5. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  6. 2022TransGeo: Transformer Is All You Need for Cross-view Image Geo-localizationcited by 166
  7. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  8. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  9. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  10. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  11. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  12. 2022Understanding and Extending Subgraph GNNs by Rethinking Their Symmetriescited by 112
38 more
  1. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  2. 2023Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisationcited by 107
  3. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  4. 2022A Generalization of ViT/MLP-Mixer to Graphscited by 93
  5. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 88
  6. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  7. 2022When Do Flat Minima Optimizers Work?cited by 78
  8. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  9. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  10. 2023A Complete Expressiveness Hierarchy for Subgraph GNNs via Subgraph Weisfeiler-Lehman Testscited by 64
  11. 2023Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimizationcited by 60
  12. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  13. 2024Friendly Sharpness-Aware Minimizationcited by 52
  14. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  15. 2022Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Modelscited by 42
  16. 2022SAM as an Optimal Relaxation of Bayescited by 40
  17. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  18. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  19. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  20. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  21. 2023ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognitioncited by 31
  22. 2024The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audiocited by 31
  23. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  24. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  25. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  26. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  27. 2024Gradient Alignment for Cross-Domain Face Anti-Spoofingcited by 26
  28. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  29. 2023Explaining Generalization Power of a DNN Using Interactive Conceptscited by 25
  30. 2024ConGeo: Robust Cross-view Geo-localization across Ground View Variationscited by 25
  31. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  32. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  33. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  34. 2023Improving Generalization of Adversarial Training via Robust Critical Fine-Tuningcited by 23
  35. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  36. 2022Adversarial Weight Perturbation Improves Generalization in Graph Neural Networkscited by 21
  37. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  38. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20

50 most cited of 293 All 293 (JSON)