mlsanityData to 7 Oct 2026

Towards Efficient and Scalable Sharpness-Aware Minimization

arXiv 2203.02714 ↗First posted 5 Mar 2022cs.LG cs.AI cs.CVcited by 129

Not on the map. Why?

Abstract (arXiv)

Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of the loss landscape and generalization, has demonstrated significant performance boosts on training large-scale models such as vision transformers. However, the update rule of SAM requires two sequential (non-parallelizable) gradient computations at each step, which can double the computational overhead. In this paper, we propose a novel algorithm LookSAM - that only periodically calculates the inner gradient ascent, to significantly reduce the additional training cost of SAM. The empirical results illustrate that LookSAM achieves similar accuracy gains to SAM while being tremendously faster - it enjoys comparable computational complexity with first-order optimizers such as SGD or Adam. To further evaluate the performance and scalability of LookSAM, we incorporate a layer-wise modification and perform experiments in the large-batch training scenario, which is more prone to converge to sharp local minima. We are the first to successfully scale up the batch size when training Vision Transformers (ViTs). With a 64k batch size, we are able to train ViTs from scratch in minutes while maintaining competitive performance.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 5 Mar 2022.

Builds on

41 of 54 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2016Wide Residual Networkscited by 3,057
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2019On the Convergence of Adam and Beyondcited by 834
  12. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
29 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  2. 2017Large Batch Training of Convolutional Networkscited by 529
  3. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  4. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  5. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  6. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  7. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  8. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  9. 2017Three Factors Influencing Minima in SGDcited by 270
  10. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  11. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  12. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  13. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  14. 2019MLPerf Training Benchmarkcited by 114
  15. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  16. 2018Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutescited by 63
  17. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
  18. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 58
  19. 2019Why are Adaptive Methods Good for Attention Models?cited by 35
  20. 2017AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networkscited by 29
  21. 2015FireCaffe: near-linear acceleration of deep neural network training on compute clusterscited by 25
  22. 2018SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learningcited by 25
  23. 2017Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutescited by 22
  24. 2018Image Classification at Supercomputer Scalecited by 16
  25. 2018On the Local Minima of the Empirical Riskcited by 12
  26. 2019Yet Another Accelerated SGD: ResNet-50 Training on ImageNet in 74.7 secondscited by 12
  27. 2019Positively Scale-Invariant Flatness of ReLU Neural Networkscited by 11
  28. 2021Concurrent Adversarial Learning for Large-Batch Trainingcited by 11
  29. 2017Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to traincited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 37; 2024: 36; 2025: 29; 2026: 14.

129 papers.

Sort:

By area:

  1. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  2. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  3. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  4. 2022Better plain ViT baselines for ImageNet-1kcited by 97
  5. 2022Sharpness-Aware Training for Freecited by 95
  6. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  7. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  8. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  9. 2024Friendly Sharpness-Aware Minimizationcited by 52
  10. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  11. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  12. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
38 more
  1. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  2. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  3. 2024Make Continual Learning Stronger via C-Flatcited by 35
  4. 2023CAME: Confidence-guided Adaptive Memory Efficient Optimizationcited by 34
  5. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  6. 2023Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimizationcited by 29
  7. 2022Escaping Saddle Points for Effective Generalization on Class-Imbalanced Datacited by 28
  8. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  9. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  10. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  11. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  12. 2024Gradient Alignment for Cross-Domain Face Anti-Spoofingcited by 26
  13. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  14. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  15. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  16. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  17. 2023FlatMatch: Bridging Labeled Data and Unlabeled Data with Cross-Sharpness for Semi-Supervised Learningcited by 20
  18. 2023Adaptive Sharpness-Aware Pruning for Robust Sparse Networkscited by 17
  19. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  20. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  21. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  22. 2022Randomized Sharpness-Aware Training for Boosting Computational Efficiency in Deep Learningcited by 14
  23. 2023Sharpness-Aware Minimization Alone can Improve Adversarial Robustnesscited by 14
  24. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  25. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  26. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  27. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  28. 2023Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalizationcited by 12
  29. 2024Domain-Inspired Sharpness-Aware Minimization Under Domain Shiftscited by 12
  30. 2023Flat Seeking Bayesian Neural Networkscited by 10
  31. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  32. 2023CR-SAM: Curvature Regularized Sharpness-Aware Minimizationcited by 10
  33. 2023Multi-Dataset Co-Training with Sharpness-Aware Optimization for Audio Anti-spoofingcited by 9
  34. 2024Switch EMA: A Free Lunch for Better Flatness and Sharpnesscited by 9
  35. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  36. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  37. 2025Beyond Magic Words: Sharpness-Aware Prompt Evolving for Robust Large Language Models with TAREcited by 9
  38. 2022Efficient Generalization Improvement Guided by Random Weight Perturbationcited by 8

50 most cited of 129 All 129 (JSON)