# SAM as an Optimal Relaxation of Bayes

arXiv: https://arxiv.org/abs/2210.01620 · HTML: https://mlsanity.com/paper/2210.01620/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 61 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [User-friendly introduction to PAC-Bayes bounds](https://mlsanity.com/paper/2110.11216/) (2021)
- [Parameter Space Noise for Exploration](https://mlsanity.com/paper/1706.01905/) (2017)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Sharpness-Aware Minimization Improves Language Model Generalization](https://mlsanity.com/paper/2110.08529/) (2021)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [Fisher SAM: Information Geometry and Sharpness Aware Minimisation](https://mlsanity.com/paper/2206.04920/) (2022)
- [Regularizing Neural Networks via Adversarial Model Perturbation](https://mlsanity.com/paper/2010.04925/) (2020)
- [Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape](https://mlsanity.com/paper/2201.08025/) (2022)
- [Deep Relaxation: partial differential equations for optimizing deep neural networks](https://mlsanity.com/paper/1704.04932/) (2017)
- [Training Binary Neural Networks using the Bayesian Learning Rule](https://mlsanity.com/paper/2002.10778/) (2020)
- [Wide Mean-Field Bayesian Neural Networks Ignore the Data](https://mlsanity.com/paper/2202.11670/) (2022)

## Built on by (most cited first, 30 of 41)

- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability](https://mlsanity.com/paper/2307.07873/) (2023)
- [Normalization Layers Are All That Sharpness-Aware Minimization Needs](https://mlsanity.com/paper/2306.04226/) (2023)
- [A Primer on Bayesian Neural Networks: Review and Debates](https://mlsanity.com/paper/2309.16314/) (2023)
- [Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima](https://mlsanity.com/paper/2306.09850/) (2023)
- [Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective](https://mlsanity.com/paper/2211.13723/) (2022)
- [Decentralized SGD and Average-direction SAM are Asymptotically Equivalent](https://mlsanity.com/paper/2306.02913/) (2023)
- [Momentum-SAM: Sharpness Aware Minimization without Computational Overhead](https://mlsanity.com/paper/2401.12033/) (2024)
- [Revisiting Random Weight Perturbation for Efficiently Improving Generalization](https://mlsanity.com/paper/2404.00357/) (2024)
- [The Interpolating Information Criterion for Overparameterized Models](https://mlsanity.com/paper/2307.07785/) (2023)
- [Uncertainty-Aware Decoding with Minimum Bayes Risk](https://mlsanity.com/paper/2503.05318/) (2025)
- [Flat Seeking Bayesian Neural Networks](https://mlsanity.com/paper/2302.02713/) (2023)
- [TRAM: Bridging Trust Regions and Sharpness Aware Minimization](https://mlsanity.com/paper/2310.03646/) (2023)
- [Entropy-MCMC: Sampling from Flat Basins with Ease](https://mlsanity.com/paper/2310.05401/) (2023)
- [Optimal Transport Model Distributional Robustness](https://mlsanity.com/paper/2306.04178/) (2023)
- [Improving Generalization with Flat Hilbert Bayesian Inference](https://mlsanity.com/paper/2410.04196/) (2024)
- [Improving SAM Requires Rethinking its Optimization Formulation](https://mlsanity.com/paper/2407.12993/) (2024)
- [The Lie-Group Bayesian Learning Rule](https://mlsanity.com/paper/2303.04397/) (2023)
- [Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Models](https://mlsanity.com/paper/2506.07247/) (2025)
- [Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2506.12409/) (2025)
- [Federated ADMM from Bayesian Duality](https://mlsanity.com/paper/2506.13150/) (2025)
- [Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers](https://mlsanity.com/paper/2603.09453/) (2026)
- [Too Sharp, Too Sure: When Calibration Follows Curvature](https://mlsanity.com/paper/2604.20614/) (2026)
- [Agnostic Sharpness-Aware Minimization](https://mlsanity.com/paper/2406.07107/) (2024)
- [Sharpness-Aware Black-Box Optimization](https://mlsanity.com/paper/2410.12457/) (2024)
- [Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization](https://mlsanity.com/paper/2505.23866/) (2025)
- [Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise](https://mlsanity.com/paper/2509.18001/) (2025)
- [SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration](https://mlsanity.com/paper/2606.10228/) (2026)
- [Flat Posterior Does Matter For Bayesian Model Averaging](https://mlsanity.com/paper/2406.15664/) (2024)
- [Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate](https://mlsanity.com/paper/2409.09984/) (2024)

## Abstract (arXiv)

Sharpness-aware minimization (SAM) and related adversarial deep-learning methods can drastically improve generalization, but their underlying mechanisms are not yet fully understood. Here, we establish SAM as a relaxation of the Bayes objective where the expected negative-loss is replaced by the optimal convex lower bound, obtained by using the so-called Fenchel biconjugate. The connection enables a new Adam-like extension of SAM to automatically obtain reasonable uncertainty estimates, while sometimes also improving its accuracy. By connecting adversarial and Bayesian methods, our work opens a new path to robustness.