mlsanityData to 14 Oct 2026

SAM as an Optimal Relaxation of Bayes

arXiv 2210.01620 ↗First posted 4 Oct 2022cs.LG cs.AI math.OCcited by 41

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware minimization (SAM) and related adversarial deep-learning methods can drastically improve generalization, but their underlying mechanisms are not yet fully understood. Here, we establish SAM as a relaxation of the Bayes objective where the expected negative-loss is replaced by the optimal convex lower bound, obtained by using the so-called Fenchel biconjugate. The connection enables a new Adam-like extension of SAM to automatically obtain reasonable uncertainty estimates, while sometimes also improving its accuracy. By connecting adversarial and Bayesian methods, our work opens a new path to robustness.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 33 of its 61 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

33 of 61 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  6. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  7. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
  8. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  10. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  11. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  12. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 308
21 more
  1. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  2. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  3. 2017Noisy Networks for Explorationcited by 266
  4. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  5. 2021User-friendly introduction to PAC-Bayes boundscited by 239
  6. 2017Parameter Space Noise for Explorationcited by 161
  7. 2019Practical Deep Learning with Bayesian Principlescited by 158
  8. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
  9. 2018Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adamcited by 131
  10. 2021Bayesian Neural Network Priors Revisitedcited by 121
  11. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  12. 2021The Bayesian Learning Rulecited by 99
  13. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  14. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  15. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  16. 2017Deep Relaxation: partial differential equations for optimizing deep neural networkscited by 27
  17. 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 23
  18. 2022Wide Mean-Field Bayesian Neural Networks Ignore the Datacited by 22
  19. 2017The proximal point method revisitedcited by 9
  20. 2017Variational Adaptive-Newton Method for Explorative Learningcited by 7
  21. 2021Lifting the Convex Conjugate in Lagrangian Relaxations: A Tractable Approach for Continuous Markov Random Fieldscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 11; 2025: 9; 2026: 8.

41 papers.

Sort:

By area:

  1. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  2. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  3. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  4. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  5. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  6. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  7. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  8. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 14
  9. 2024Revisiting Random Weight Perturbation for Efficiently Improving Generalizationcited by 13
  10. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 12
  11. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 11
  12. 2023Flat Seeking Bayesian Neural Networkscited by 10
29 more
  1. 2023TRAM: Bridging Trust Regions and Sharpness Aware Minimizationcited by 6
  2. 2023Entropy-MCMC: Sampling from Flat Basins with Easecited by 6
  3. 2023Optimal Transport Model Distributional Robustnesscited by 5
  4. 2024Improving Generalization with Flat Hilbert Bayesian Inferencecited by 5
  5. 2024Improving SAM Requires Rethinking its Optimization Formulationcited by 4
  6. 2023The Lie-Group Bayesian Learning Rulecited by 3
  7. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  8. 2025Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Modelscited by 3
  9. 2025Federated ADMM from Bayesian Dualitycited by 3
  10. 2026Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformerscited by 3
  11. 2026Too Sharp, Too Sure: When Calibration Follows Curvaturecited by 3
  12. 2024Agnostic Sharpness-Aware Minimizationcited by 2
  13. 2024Sharpness-Aware Black-Box Optimizationcited by 2
  14. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  15. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  16. 2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
  17. 2024Flat Posterior Does Matter For Bayesian Model Averagingcited by 1
  18. 2024Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Ratecited by 1
  19. 2026Parameter Exploration for RLVR via Variational Learningcited by 1
  20. 2023Sharpness-Aware Teleportation on Riemannian Manifoldscited by 0
  21. 2024Flatness Improves Backbone Generalisation in Few-shot Classificationcited by 0
  22. 2024Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamicscited by 0
  23. 2025Variational Learning Induces Adaptive Label Smoothingcited by 0
  24. 2025A note on the Artstein-Avidan-Milman's generalized Legendre transformscited by 0
  25. 2025Sharpness-Aware Minimization Can Hallucinate Minimizerscited by 0
  26. 2026Sharpness-Aware Hybrid Model Learning for Architecture-Agnostic Parameter Estimationcited by 0
  27. 2026Incorruptible Neural Networks: Training Models that can Generalize to Large Internal Perturbationscited by 0
  28. 2026OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizerscited by 0
  29. 2026A Generalization of Amari's Bayesian Dualitycited by 0