mlsanityData to 7 Oct 2026

Sharpness-Aware Minimization for Efficiently Improving Generalization

arXiv 2010.01412 ↗First posted 3 Oct 2020cs.LG stat.MLcited by 1,404

Not on the map. Why?

Cited by 40 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-10, CIFAR-100, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels. We open source our code at urlhttps://github.com/google-research/sam.

Key facts. Cited by 1,404 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 104; 2022: 175; 2023: 256; 2024: 305; 2025: 303; 2026: 261.

1,404 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2021Transformers in Vision: A Surveycited by 1,009
  5. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  6. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  7. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  8. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  9. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  10. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  11. 2022Detecting Deepfakes with Self-Blended Imagescited by 332
  12. 2022How Do Vision Transformers Work?cited by 305
38 more
  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  4. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  5. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  6. 2021Vision Transformers are Robust Learnerscited by 208
  7. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  8. 2021ResViT: Residual vision transformers for multi-modal medical image synthesiscited by 177
  9. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  10. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  11. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  12. 2021VOLO: Vision Outlooker for Visual Recognitioncited by 164
  13. 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
  14. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  15. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  16. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  17. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  18. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  19. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  20. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  21. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  22. 2021Scaling Up Influence Functionscited by 130
  23. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  24. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  25. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  26. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  27. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  28. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  29. 2021Cycle Self-Training for Domain Adaptationcited by 120
  30. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  31. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  32. 2023Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisationcited by 107
  33. 2021On the Adversarial Robustness of Vision Transformerscited by 104
  34. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  35. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  36. 2021The Efficiency Misnomercited by 98
  37. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  38. 2022Sharpness-Aware Training for Freecited by 95

50 most cited of 1,404 All 1,404 (JSON)