# Structured Partial Stochasticity in Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2405.17666 · HTML: https://mlsanity.com/paper/2405.17666/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 13 of 13)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Bayesian Deep Learning via Subnetwork Inference](https://mlsanity.com/paper/2010.14689/) (2020)
- [Do Bayesian Neural Networks Need To Be Fully Stochastic?](https://mlsanity.com/paper/2211.06291/) (2022)
- [Benchmarking the Neural Linear Model for Regression](https://mlsanity.com/paper/1912.08416/) (2019)
- [Wide Mean-Field Bayesian Neural Networks Ignore the Data](https://mlsanity.com/paper/2202.11670/) (2022)
- [Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry](https://mlsanity.com/paper/2304.02902/) (2023)
- [On the detrimental effect of invariances in the likelihood for variational inference](https://mlsanity.com/paper/2209.07157/) (2022)
- [A Compact Representation for Bayesian Neural Networks By Removing Permutation Symmetry](https://mlsanity.com/paper/2401.00611/) (2023)

## Abstract (arXiv)

Bayesian neural network posterior distributions have a great number of modes that correspond to the same network function. The abundance of such modes can make it difficult for approximate inference methods to do their job. Recent work has demonstrated the benefits of partial stochasticity for approximate inference in Bayesian neural networks; inference can be less costly and performance can sometimes be improved. I propose a structured way to select the deterministic subset of weights that removes neuron permutation symmetries, and therefore the corresponding redundant posterior modes. With a drastically simplified posterior distribution, the performance of existing approximate inference schemes is found to be greatly improved.