# A Simple Baseline for Bayesian Uncertainty in Deep Learning

arXiv: https://arxiv.org/abs/1902.02476 · HTML: https://mlsanity.com/paper/1902.02476/

## Key facts

- Cited by 560 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2019.

## Built on by (most cited first, 30 of 560)

- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06329/) (2021)
- [Deep Deterministic Uncertainty: A Simple Baseline](https://mlsanity.com/paper/2102.11582/) (2021)
- [DICE: Leveraging Sparsification for Out-of-Distribution Detection](https://mlsanity.com/paper/2111.09805/) (2021)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [Non-Parametric Outlier Synthesis](https://mlsanity.com/paper/2303.02966/) (2023)
- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning](https://mlsanity.com/paper/2104.04975/) (2021)
- [DEUP: Direct Epistemic Uncertainty Prediction](https://mlsanity.com/paper/2102.08501/) (2021)
- [The Bayesian Learning Rule](https://mlsanity.com/paper/2107.04562/) (2021)
- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [Model Stock: All we need is just a few fine-tuned models](https://mlsanity.com/paper/2403.19522/) (2024)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Learning Neural Network Subspaces](https://mlsanity.com/paper/2102.10472/) (2021)
- [Stochastic Neural Radiance Fields: Quantifying Uncertainty in Implicit 3D Representations](https://mlsanity.com/paper/2109.02123/) (2021)
- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification](https://mlsanity.com/paper/2110.14012/) (2021)
- [Dream the Impossible: Outlier Imagination with Diffusion Models](https://mlsanity.com/paper/2309.13415/) (2023)
- [Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling](https://mlsanity.com/paper/2102.13042/) (2021)
- [Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review](https://mlsanity.com/paper/2303.10462/) (2023)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Fair Conformal Predictors for Applications in Medical Imaging](https://mlsanity.com/paper/2109.04392/) (2021)
- [Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences](https://mlsanity.com/paper/2002.09089/) (2020)
- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Personalized Federated Learning via Variational Bayesian Inference](https://mlsanity.com/paper/2206.07977/) (2022)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)

## Abstract (arXiv)

We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning. Stochastic Weight Averaging (SWA), which computes the first moment of stochastic gradient descent (SGD) iterates with a modified learning rate schedule, has recently been shown to improve generalization in deep learning. With SWAG, we fit a Gaussian using the SWA solution as the first moment and a low rank plus diagonal covariance also derived from the SGD iterates, forming an approximate posterior distribution over neural network weights; we then sample from this Gaussian distribution to perform Bayesian model averaging. We empirically find that SWAG approximates the shape of the true posterior, in accordance with results describing the stationary distribution of SGD iterates. Moreover, we demonstrate that SWAG performs well on a wide variety of tasks, including out of sample detection, calibration, and transfer learning, in comparison to many popular alternatives including MC dropout, KFAC Laplace, SGLD, and temperature scaling.