# Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty

arXiv: https://arxiv.org/abs/2305.00934 · HTML: https://mlsanity.com/paper/2305.00934/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 86 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Multiplicative Normalizing Flows for Variational Bayesian Neural Networks](https://mlsanity.com/paper/1703.01961/) (2017)
- [Bayesian Compression for Deep Learning](https://mlsanity.com/paper/1705.08665/) (2017)
- [Hierarchical Variational Models](https://mlsanity.com/paper/1511.02386/) (2015)
- [Model Selection in Bayesian Neural Networks via Horseshoe Priors](https://mlsanity.com/paper/1705.10388/) (2017)
- [On Markov chain Monte Carlo methods for tall data](https://mlsanity.com/paper/1505.02827/) (2015)
- [Posterior Concentration for Sparse Deep Learning](https://mlsanity.com/paper/1803.09138/) (2018)
- [Structured Bayesian Pruning via Log-Normal Multiplicative Noise](https://mlsanity.com/paper/1705.07283/) (2017)
- [Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee](https://mlsanity.com/paper/2011.07439/) (2020)
- [Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors](https://mlsanity.com/paper/1806.05975/) (2018)
- [Convergence Rates of Variational Inference in Sparse Deep Learning](https://mlsanity.com/paper/1908.04847/) (2019)
- [Austerity in MCMC Land: Cutting the Metropolis-Hastings Budget](https://mlsanity.com/paper/1304.5299/) (2013)
- [Firefly Monte Carlo: Exact MCMC with Subsets of Data](https://mlsanity.com/paper/1403.5693/) (2014)
- [Speeding Up MCMC by Efficient Data Subsampling](https://mlsanity.com/paper/1404.4178/) (2014)
- [Variational Inference to Measure Model Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/1902.10189/) (2019)
- [Variance Networks: When Expectation Does Not Meet Your Expectations](https://mlsanity.com/paper/1803.03764/) (2018)
- [Expectation Propagation for Neural Networks with Sparsity-promoting Priors](https://mlsanity.com/paper/1303.6938/) (2013)
- [Combining Model and Parameter Uncertainty in Bayesian Neural Networks](https://mlsanity.com/paper/1903.07594/) (2019)
- [Model Averaging and its Use in Economics](https://mlsanity.com/paper/1709.08221/) (2017)
- [Flexible Bayesian Nonlinear Model Configuration](https://mlsanity.com/paper/2003.02929/) (2020)
- [Statistical Foundation of Variational Bayes Neural Networks](https://mlsanity.com/paper/2006.15786/) (2020)
- [Mixtures of g-priors in Generalized Linear Models](https://mlsanity.com/paper/1503.06913/) (2015)

## Built on by (most cited first, 5 of 5)

- [Sparsifying Bayesian neural networks with latent binary variables and normalizing flows](https://mlsanity.com/paper/2305.03395/) (2023)
- [BMRS: Bayesian Model Reduction for Structured Pruning](https://mlsanity.com/paper/2406.01345/) (2024)
- [Bayesian sparsification for deep neural networks with Bayesian model reduction](https://mlsanity.com/paper/2309.12095/) (2023)
- [Variational Bayes for Federated Continual Learning](https://mlsanity.com/paper/2405.14291/) (2024)
- [RF-BayesPhysNet: A Bayesian rPPG Uncertainty Estimation Method for Complex Scenarios](https://mlsanity.com/paper/2504.03915/) (2025)

## Abstract (arXiv)

Bayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: Parameter and prediction uncertainties become easily available, facilitating rigorous statistical analysis. Furthermore, prior knowledge can be incorporated. However, so far, there have been no scalable techniques capable of combining both structural and parameter uncertainty. In this paper, we apply the concept of model uncertainty as a framework for structural learning in BNNs and hence make inference in the joint space of structures/models and parameters. Moreover, we suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities to incorporate the model space constraints. Experimental results on a range of benchmark datasets show that we obtain comparable accuracy results with the competing models, but based on methods that are much more sparse than ordinary BNNs.