# Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise

arXiv: https://arxiv.org/abs/2103.08497 · HTML: https://mlsanity.com/paper/2103.08497/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 15 Mar 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [Stochastic Gradient Descent as Approximate Bayesian Inference](https://mlsanity.com/paper/1704.04289/) (2017)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Practical Deep Learning with Bayesian Principles](https://mlsanity.com/paper/1906.02506/) (2019)
- [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017)
- [Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches](https://mlsanity.com/paper/1803.04386/) (2018)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [Lightweight Probabilistic Deep Networks](https://mlsanity.com/paper/1805.11327/) (2018)
- [Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation](https://mlsanity.com/paper/1908.00598/) (2019)
- [Natural-Parameter Networks: A Class of Probabilistic Neural Networks](https://mlsanity.com/paper/1611.00448/) (2016)
- [Stochastic Expectation Propagation](https://mlsanity.com/paper/1506.04132/) (2015)
- [Variational Bayesian dropout: pitfalls and fixes](https://mlsanity.com/paper/1807.01969/) (2018)
- [Bayesian Inference for Large Scale Image Classification](https://mlsanity.com/paper/1908.03491/) (2019)
- [Robust Convolutional Neural Networks under Adversarial Noise](https://mlsanity.com/paper/1511.06306/) (2015)
- [Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation](https://mlsanity.com/paper/1805.07654/) (2018)
- [Closed Form Variational Objectives For Bayesian Neural Networks with a Single Hidden Layer](https://mlsanity.com/paper/1811.00686/) (2018)

## Built on by (most cited first, 2 of 2)

- [Propagating Variational Model Uncertainty for Bioacoustic Call Label Smoothing](https://mlsanity.com/paper/2210.10526/) (2022)
- [Probabilistic computation and uncertainty quantification with emerging covariance](https://mlsanity.com/paper/2305.19265/) (2023)

## Abstract (arXiv)

To adopt neural networks in safety critical domains, knowing whether we can trust their predictions is crucial. Bayesian neural networks (BNNs) provide uncertainty estimates by averaging predictions with respect to the posterior weight distribution. Variational inference methods for BNNs approximate the intractable weight posterior with a tractable distribution, yet mostly rely on sampling from the variational distribution during training and inference. Recent sampling-free approaches offer an alternative, but incur a significant parameter overhead. We here propose a more efficient parameterization of the posterior approximation for sampling-free variational inference that relies on the distribution induced by multiplicative Gaussian activation noise. This allows us to combine parameter efficiency with the benefits of sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks including ImageNet.