mlsanityData to 7 Oct 2026

Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning

arXiv 2405.20439 ↗First posted 30 May 2024cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) has emerged as a promising alternative optimizer to stochastic gradient descent (SGD). The originally-proposed motivation behind SAM was to bias neural networks towards flatter minima that are believed to generalize better. However, recent studies have shown conflicting evidence on the relationship between flatness and generalization, suggesting that flatness does fully explain SAM's success. Sidestepping this debate, we identify an orthogonal effect of SAM that is beneficial out-of-distribution: we argue that SAM implicitly balances the quality of diverse features. SAM achieves this effect by adaptively suppressing well-learned features which gives remaining features opportunity to be learned. We show that this mechanism is beneficial in datasets that contain redundant or spurious features where SGD falls for the simplicity bias and would not otherwise learn all available features. Our insights are supported by experiments on real data: we demonstrate that SAM improves the quality of features in datasets containing redundant or spurious features, including CelebA, Waterbirds, CIFAR-MNIST, and DomainBed.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 72 of its 79 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

72 of 79 references.

References · 72

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  7. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2018On the Spectral Bias of Neural Networkscited by 1,321
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  12. 2020In Search of Lost Domain Generalizationcited by 1,042
38 more
  1. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  2. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  3. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  4. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  5. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  6. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  7. 2018Do Better ImageNet Models Transfer Better?cited by 560
  8. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  12. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  13. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  14. 2020What is being transferred in transfer learning?cited by 365
  15. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  16. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  17. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  18. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  19. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  20. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  21. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  22. 2016What makes ImageNet good for transfer learning?cited by 192
  23. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  24. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  25. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  26. 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
  27. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  28. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  29. 2020What shapes feature representations? Exploring datasets, architectures, and trainingcited by 125
  30. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  31. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  32. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  33. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  34. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  35. 2022Sharpness-Aware Training for Freecited by 95
  36. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  37. 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
  38. 2022When Do Flat Minima Optimizers Work?cited by 78

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 2.

9 papers.

  1. 2024Understanding Visual Feature Reliance through the Lens of Complexitycited by 15
  2. 2025Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Modelscited by 2
  3. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  4. 2025A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Biascited by 2
  5. 2024Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Ratecited by 1
  6. 2024Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalizationcited by 1
  7. 2025SAMO: A Lightweight Sharpness-Aware Approach for Multi-Task Optimization with Joint Global-Local Perturbationcited by 1
  8. 2026Easy to Learn, Yet Hard to Forget: Towards Robust Unlearning Under Biascited by 1
  9. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0