mlsanityData to 7 Oct 2026

Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term

arXiv 2305.15817 ↗First posted 25 May 2023cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Deep Neural Networks (DNNs) generalization is known to be closely related to the flatness of minima, leading to the development of Sharpness-Aware Minimization (SAM) for seeking flatter minima and better generalization. In this paper, we revisit the loss of SAM and propose a more general method, called WSAM, by incorporating sharpness as a regularization term. We prove its generalization bound through the combination of PAC and Bayes-PAC techniques, and evaluate its performance on various public datasets. The results demonstrate that WSAM achieves improved generalization, or is at least highly competitive, compared to the vanilla optimizer, SAM and its variants. The code is available at https://github.com/intelligent-machine-learning/atorch/tree/main/atorch/optimizers.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

24 of 35 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2016Wide Residual Networkscited by 3,057
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  12. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
12 more
  1. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  2. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  3. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
  4. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  5. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  6. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  7. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  8. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  9. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  10. 2022When Do Flat Minima Optimizers Work?cited by 78
  11. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  12. 2019Understanding Generalization through Visualizationscited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 3.

16 papers.

Sort:

  1. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  2. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  3. 2024FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustnesscited by 5
  4. 2024Do Sharpness-based Optimizers Improve Generalization in Medical Image Analysis?cited by 3
  5. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
  6. 2025Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Modelscited by 3
  7. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  8. 2025Model Merging via Multi-Teacher Knowledge Distillationcited by 2
  9. 2025TimeCF: A TimeMixer-Based Model with adaptive Convolution and Sharpness-Aware Minimization Frequency Domain Loss for long-term time seris forecastingcited by 1
  10. 2025Towards Better Generalization via Distributional Input Projection Networkcited by 1
  11. 2025LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimizationcited by 0
  12. 2025Do Flat Minima Improve Sparse Novel View Synthesis?cited by 0
4 more
  1. 2025Sharpness-aware Federated Graph Learningcited by 0
  2. 2026X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correctioncited by 0
  3. 2026Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementationcited by 0
  4. 2026Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimizationcited by 0