# Single-shot Bayesian approximation for neural networks

arXiv: https://arxiv.org/abs/2308.12785 · HTML: https://mlsanity.com/paper/2308.12785/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 28 of 28)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision](https://mlsanity.com/paper/1906.01620/) (2019)
- [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017)
- [A General Framework for Uncertainty Estimation in Deep Learning](https://mlsanity.com/paper/1907.06890/) (2019)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [The Limits and Potentials of Deep Learning for Robotics](https://mlsanity.com/paper/1804.06557/) (2018)
- [Uncertainty in Neural Networks: Approximately Bayesian Ensembling](https://mlsanity.com/paper/1810.05546/) (2018)
- [Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection](https://mlsanity.com/paper/1804.05132/) (2018)
- [Lightweight Probabilistic Deep Networks](https://mlsanity.com/paper/1805.11327/) (2018)
- [Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation](https://mlsanity.com/paper/1908.00598/) (2019)
- [BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors](https://mlsanity.com/paper/1903.03838/) (2019)
- [HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentation](https://mlsanity.com/paper/1804.02967/) (2018)
- [Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control](https://mlsanity.com/paper/1811.06817/) (2018)
- [Natural-Parameter Networks: A Class of Probabilistic Neural Networks](https://mlsanity.com/paper/1611.00448/) (2016)
- [Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables](https://mlsanity.com/paper/1703.00091/) (2017)
- [Deep transformation models: Tackling complex regression problems with neural network based transformation models](https://mlsanity.com/paper/2004.00464/) (2020)
- [Robust Convolutional Neural Networks under Adversarial Noise](https://mlsanity.com/paper/1511.06306/) (2015)
- [Integrating uncertainty in deep neural networks for MRI based stroke analysis](https://mlsanity.com/paper/2008.06332/) (2020)
- [Single Shot MC Dropout Approximation](https://mlsanity.com/paper/2007.03293/) (2020)
- [Deep and interpretable regression models for ordinal outcomes](https://mlsanity.com/paper/2010.08376/) (2020)

## Abstract (arXiv)

Deep neural networks (NNs) are known for their high-prediction performances. However, NNs are prone to yield unreliable predictions when encountering completely new situations without indicating their uncertainty. Bayesian variants of NNs (BNNs), such as Monte Carlo (MC) dropout BNNs, do provide uncertainty measures and simultaneously increase the prediction performance. The only disadvantage of BNNs is their higher computation time during test time because they rely on a sampling approach. Here we present a single-shot MC dropout approximation that preserves the advantages of BNNs while being as fast as NNs. Our approach is based on moment propagation (MP) and allows to analytically approximate the expected value and the variance of the MC dropout signal for commonly used layers in NNs, i.e. convolution, max pooling, dense, softmax, and dropout layers. The MP approach can convert an NN into a BNN without re-training given the NN has been trained with standard dropout. We evaluate our approach on different benchmark datasets and a simulated toy example in a classification and regression setting. We demonstrate that our single-shot MC dropout approximation resembles the point estimate and the uncertainty estimate of the predictive distribution that is achieved with an MC approach, while being fast enough for real-time deployments of BNNs. We show that using part of the saved time to combine our MP approach with deep ensemble techniques does further improve the uncertainty measures.