mlsanityData to 7 Oct 2026

Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithm

arXiv 2501.06603 ↗First posted 11 Jan 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Targeting solutions over `flat' regions of the loss landscape, sharpness-aware minimization (SAM) has emerged as a powerful tool to improve generalizability of deep neural network based learning. While several SAM variants have been developed to this end, a unifying approach that also guides principled algorithm design has been elusive. This contribution leverages preconditioning (pre) to unify SAM variants and provide not only unifying convergence analysis, but also valuable insights. Building upon preSAM, a novel algorithm termed infoSAM is introduced to address the so-called adversarial model degradation issue in SAM by adjusting gradients depending on noise estimates. Extensive numerical tests demonstrate the superiority of infoSAM across various benchmarks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 35 references that are in the map. First posted on arXiv 11 Jan 2025.

Builds on

26 of 35 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2016Wide Residual Networkscited by 3,057
  5. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  9. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  10. 2021AST: Audio Spectrogram Transformercited by 634
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  12. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
14 more
  1. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  2. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  3. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  4. 2016Deep Pyramidal Residual Networkscited by 163
  5. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  6. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  7. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  8. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  9. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  10. 2022The Effects of Regularization and Data Augmentation are Class Dependentcited by 67
  11. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  12. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  13. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  14. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Flatness and Gradient Alignment Are Both Necessary: Spectral-Aware Gradient-Aligned Exploration for Multi-Distribution Learningcited by 0