mlsanityData to 7 Oct 2026

Enhancing Sharpness-Aware Optimization Through Variance Suppression

arXiv 2309.15639 ↗First posted 27 Sep 2023cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware minimization (SAM) has well documented merits in enhancing generalization of deep neural networks, even without sizable data augmentation. Embracing the geometry of the loss function, where neighborhoods of 'flat minima' heighten generalization ability, SAM seeks 'flat valleys' by minimizing the maximum loss caused by an adversary perturbing parameters within the neighborhood. Although critical to account for sharpness of the loss function, such an 'over-friendly adversary' can curtail the outmost level of generalization. The novel approach of this contribution fosters stabilization of adversaries through variance suppression (VaSSO) to avoid such friendliness. VaSSO's provable stability safeguards its numerical improvement over SAM in model-agnostic tasks, including image classification and machine translation. In addition, experiments confirm that VaSSO endows SAM with robustness against high levels of label noise.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 44 of its 57 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

44 of 57 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  7. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  9. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  10. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  11. 2017Exploring Generalization in Deep Learningcited by 595
  12. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
32 more
  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  2. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  3. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  4. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  5. 2017Three Factors Influencing Minima in SGDcited by 270
  6. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  7. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  8. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  9. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  10. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
  11. 2020Implicit Gradient Regularizationcited by 145
  12. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  13. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  14. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  15. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  16. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  17. 2022Sharpness-Aware Training for Freecited by 95
  18. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  19. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  20. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  21. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  22. 2018Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximizationcited by 47
  23. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  24. 2016Stochastic Frank-Wolfe Methods for Nonconvex Optimizationcited by 40
  25. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  26. 2020Understanding and Increasing Efficiency of Frank-Wolfe Adversarial Trainingcited by 14
  27. 2022GA-SAM: Gradient-Strength based Adaptive Sharpness-Aware Minimization for Improved Generalizationcited by 13
  28. 2020Accelerated Stochastic Gradient-free and Projection-free Methodscited by 12
  29. 2019On the Convergence of SARAH and Beyondcited by 10
  30. 2023Exploring the Effect of Multi-step Ascent in Sharpness-Aware Minimizationcited by 9
  31. 2019Almost Tune-Free Variance Reductioncited by 7
  32. 2021Heavy Ball Momentum for Conditional Gradientcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 14; 2026: 7.

37 papers.

Sort:

By area:

  1. 2024Friendly Sharpness-Aware Minimizationcited by 52
  2. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  3. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  4. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  5. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  6. 2024Domain-Inspired Sharpness-Aware Minimization Under Domain Shiftscited by 12
  7. 2025Sharpness-Aware Minimization: General Analysis and Improved Ratescited by 10
  8. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  9. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Modelscited by 9
  10. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  11. 2024SAMPa: Sharpness-aware Minimization Parallelizedcited by 8
  12. 2024Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Modelscited by 7
25 more
  1. 2024Tilted Sharpness-Aware Minimizationcited by 6
  2. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  3. 2025Towards Robust Influence Functions with Flat Validation Minimacited by 4
  4. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  5. 2025Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Modelscited by 3
  6. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 3
  7. 2024GAQAT: gradient-adaptive quantization-aware training for domain generalizationcited by 2
  8. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  9. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  10. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  11. 2024Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Ratecited by 1
  12. 2024Sharpness-Aware Minimization with Adaptive Regularization for Training Deep Neural Networkscited by 1
  13. 2025Preconditioned Sharpness-Aware Minimization: Unifying Analysis and a Novel Learning Algorithmcited by 1
  14. 2025VASSO: Variance Suppression for Sharpness-Aware Minimizationcited by 1
  15. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1
  16. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
  17. 2024Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awarenesscited by 0
  18. 2024Understanding SAM's Robustness to Noisy Labels through Gradient Down-weightingcited by 0
  19. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
  20. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  21. 2025Sharpness-aware Federated Graph Learningcited by 0
  22. 2026Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementationcited by 0
  23. 2026Flatness and Gradient Alignment Are Both Necessary: Spectral-Aware Gradient-Aligned Exploration for Multi-Distribution Learningcited by 0
  24. 2026FedVSSAM: Mitigating Flatness Incompatibility in Sharpness-Aware Federated Learningcited by 0
  25. 2026Navigating Potholes with Geometry-Aware Sharpness Minimizationcited by 0