mlsanityData to 14 Oct 2026

Sharpness-Aware Minimization with Adaptive Regularization for Training Deep Neural Networks

arXiv 2412.16854 ↗First posted 22 Dec 2024cs.LG cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) has proven highly effective in improving model generalization in machine learning tasks. However, SAM employs a fixed hyperparameter associated with the regularization to characterize the sharpness of the model. Despite its success, research on adaptive regularization methods based on SAM remains scarce. In this paper, we propose the SAM with Adaptive Regularization (SAMAR), which introduces a flexible sharpness ratio rule to update the regularization parameter dynamically. We provide theoretical proof of the convergence of SAMAR for functions satisfying the Lipschitz continuity. Additionally, experiments on image recognition tasks using CIFAR-10 and CIFAR-100 demonstrate that SAMAR enhances accuracy and model generalization.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 22 Dec 2024.

Builds on

21 of 28 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Wide Residual Networkscited by 3,070
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  7. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  8. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  9. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  10. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  11. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  12. 2017Sub-sampled Cubic Regularization for Non-convex Optimizationcited by 63
9 more
  1. 2024Friendly Sharpness-Aware Minimizationcited by 53
  2. 2019Understanding the Role of Momentum in Stochastic Gradient Methodscited by 44
  3. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  4. 2020A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performancecited by 26
  5. 2023AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networkscited by 21
  6. 2018Adaptive regularization with cubics on manifoldscited by 17
  7. 2024Fundamental Convergence Analysis of Sharpness-Aware Minimizationcited by 14
  8. 2021Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimizationcited by 8
  9. 2019General Proximal Incremental Aggregated Gradient Algorithms: Better and Novel Results under General Schemecited by 6

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025LightSAM: Parameter-Agnostic Sharpness-Aware Minimizationcited by 3