mlsanityData to 7 Oct 2026

Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

arXiv 2210.05177 ↗First posted 11 Oct 2022cs.LG cs.AI cs.CVcited by 70

Not on the map. Why?

Abstract (arXiv)

Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation, i.e., double the overhead of common optimizers like Stochastic Gradient Descent (SGD). In this paper, we propose an efficient and effective training scheme coined as Sparse SAM (SSAM), which achieves sparse perturbation by a binary mask. To obtain the sparse mask, we provide two solutions which are based onFisher information and dynamic sparse training, respectively. In addition, we theoretically prove that SSAM can converge at the same rate as SAM, i.e., O(log T/√T). Sparse SAM not only has the potential for training acceleration but also smooths the loss landscape effectively. Extensive experimental results on CIFAR10, CIFAR100, and ImageNet-1K confirm the superior efficiency of our method to SAM, and the performance is preserved or even better with a perturbation of merely 50% sparsity. Code is availiable at https://github.com/Mi-Peng/Sparse-Sharpness-Aware-Minimization.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 44 of its 64 references that are in the map. First posted on arXiv 11 Oct 2022.

Builds on

44 of 64 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2016Wide Residual Networkscited by 3,057
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
32 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  13. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  14. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  15. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  16. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  17. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  18. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  19. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  20. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  21. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  22. 2018Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasiscited by 143
  23. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  24. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  25. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  26. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  27. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  28. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  29. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  30. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  31. 2021Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Accelerationcited by 25
  32. 2021Concurrent Adversarial Learning for Large-Batch Trainingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 22; 2024: 24; 2025: 14; 2026: 9.

70 papers.

Sort:

By area:

  1. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  2. 2023Enhancing Fine-Tuning Based Backdoor Defense with Sharpness-Aware Minimizationcited by 60
  3. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  4. 2024Friendly Sharpness-Aware Minimizationcited by 52
  5. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  6. 2023Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscapecited by 46
  7. 2023Improving the Model Consistency of Decentralized Federated Learningcited by 45
  8. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  9. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  10. 2023Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimizationcited by 29
  11. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  12. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
38 more
  1. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  2. 2025The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Trainingcited by 27
  3. 2023The Crucial Role of Normalization in Sharpness-Aware Minimizationcited by 26
  4. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  5. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  6. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  7. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  8. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  9. 2024Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognitioncited by 15
  10. 2023Sharpness-Aware Minimization Alone can Improve Adversarial Robustnesscited by 14
  11. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  12. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  13. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  14. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  15. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  16. 2024Domain-Inspired Sharpness-Aware Minimization Under Domain Shiftscited by 12
  17. 2023CR-SAM: Curvature Regularized Sharpness-Aware Minimizationcited by 10
  18. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
  19. 2025Sharpness-Aware Minimization: General Analysis and Improved Ratescited by 10
  20. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  21. 2022Efficient Generalization Improvement Guided by Random Weight Perturbationcited by 8
  22. 2023Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Trainingcited by 8
  23. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  24. 2024Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Modelscited by 8
  25. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  26. 2024Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minimacited by 6
  27. 2024Bi-TTA: Bidirectional Test-Time Adapter for Remote Physiological Measurementcited by 6
  28. 2024Tilted Sharpness-Aware Minimizationcited by 6
  29. 2023Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacycited by 5
  30. 2023Asymmetrically Decentralized Federated Learningcited by 5
  31. 2023Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizercited by 4
  32. 2023Critical Influence of Overparameterization on Sharpness-aware Minimizationcited by 3
  33. 2024SSE-SAM: Balancing Head and Tail Classes Gradually through Stage-Wise SAMcited by 3
  34. 2025Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learningcited by 3
  35. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 3
  36. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 3
  37. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 2
  38. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2

50 most cited of 70 All 70 (JSON)