mlsanityData to 14 Oct 2026

Rethinking Sharpness-Aware Minimization as Variational Inference

arXiv 2210.10452 ↗First posted 19 Oct 2022stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Sharpness-aware minimization (SAM) aims to improve the generalisation of gradient-based learning by seeking out flat minima. In this work, we establish connections between SAM and Mean-Field Variational Inference (MFVI) of neural network parameters. We show that both these methods have interpretations as optimizing notions of flatness, and when using the reparametrisation trick, they both boil down to calculating the gradient at a perturbed version of the current mean parameter. This thinking motivates our study of algorithms that combine or interpolate between SAM and MFVI. We evaluate the proposed variational algorithms on several benchmark datasets, and compare their performance to variants of SAM. Taking a broader perspective, our work suggests that SAM-like updates can be used as a drop-in replacement for the reparametrisation trick.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

15 of 24 references.

References · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2016Wide Residual Networkscited by 3,070
  4. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  6. 2019When Does Label Smoothing Help?cited by 988
  7. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  8. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  9. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  10. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  11. 2021On the Origin of Implicit Regularization in Stochastic Gradient Descentcited by 184
  12. 2020Implicit Gradient Regularizationcited by 146
3 more
  1. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  2. 2018Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameterscited by 58
  3. 2021SGD Implicitly Regularizes Generalization Errorcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 0; 2026: 1.

8 papers.

  1. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  2. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  3. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  4. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  5. 2023On Statistical Properties of Sharpness-Aware Minimization: Provable Guaranteescited by 5
  6. 2024A Precise Characterization of SGD Stability Using Loss Surface Geometrycited by 2
  7. 2026Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMscited by 1
  8. 2024Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamicscited by 0