# On the detrimental effect of invariances in the likelihood for variational inference

arXiv: https://arxiv.org/abs/2209.07157 · HTML: https://mlsanity.com/paper/2209.07157/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Builds on (most cited first, 19 of 19)

- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Variational Continual Learning](https://mlsanity.com/paper/1710.10628/) (2017)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors](https://mlsanity.com/paper/2005.07186/) (2020)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape](https://mlsanity.com/paper/1907.02911/) (2019)
- [All You Need is a Good Functional Prior for Bayesian Deep Learning](https://mlsanity.com/paper/2011.12829/) (2020)
- [Variational Implicit Processes](https://mlsanity.com/paper/1806.02390/) (2018)
- [Function Space Particle Optimization for Bayesian Neural Networks](https://mlsanity.com/paper/1902.09754/) (2019)
- [Understanding Variational Inference in Function-Space](https://mlsanity.com/paper/2011.09421/) (2020)
- [Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors](https://mlsanity.com/paper/1806.05975/) (2018)
- [SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient](https://mlsanity.com/paper/1811.04504/) (2018)
- [Deep kernel processes](https://mlsanity.com/paper/2010.01590/) (2020)
- [Overpruning in Variational Bayesian Neural Networks](https://mlsanity.com/paper/1801.06230/) (2018)
- [The prior can generally only be understood in the context of the likelihood](https://mlsanity.com/paper/1708.07487/) (2017)
- [Wide Mean-Field Variational Bayesian Neural Networks Ignore the Data](https://mlsanity.com/paper/2106.07052/) (2021)
- [Detecting Parameter Symmetries in Probabilistic Models](https://mlsanity.com/paper/1312.5386/) (2013)

## Built on by (most cited first, 8 of 8)

- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks](https://mlsanity.com/paper/2408.05496/) (2024)
- [On permutation symmetries in Bayesian neural network posteriors: a variational perspective](https://mlsanity.com/paper/2310.10171/) (2023)
- [Improving the performance of Stein variational inference through extreme sparsification of physically-constrained neural network models](https://mlsanity.com/paper/2407.00761/) (2024)
- [Variational Bayesian Neural Networks via Resolution of Singularities](https://mlsanity.com/paper/2302.06035/) (2023)
- [The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE](https://mlsanity.com/paper/2502.17391/) (2025)
- [On the Approximation and Complexity of Deep Neural Networks to Invariant Functions](https://mlsanity.com/paper/2210.15279/) (2022)
- [Structured Partial Stochasticity in Bayesian Neural Networks](https://mlsanity.com/paper/2405.17666/) (2024)

## Abstract (arXiv)

Variational Bayesian posterior inference often requires simplifying approximations such as mean-field parametrisation to ensure tractability. However, prior work has associated the variational mean-field approximation for Bayesian neural networks with underfitting in the case of small datasets or large model sizes. In this work, we show that invariances in the likelihood function of over-parametrised models contribute to this phenomenon because these invariances complicate the structure of the posterior by introducing discrete and/or continuous modes which cannot be well approximated by Gaussian mean-field distributions. In particular, we show that the mean-field approximation has an additional gap in the evidence lower bound compared to a purpose-built posterior that takes into account the known invariances. Importantly, this invariance gap is not constant; it vanishes as the approximation reverts to the prior. We proceed by first considering translation invariances in a linear model with a single data point in detail. We show that, while the true posterior can be constructed from a mean-field parametrisation, this is achieved only if the objective function takes into account the invariance gap. Then, we transfer our analysis of the linear model to neural networks. Our analysis provides a framework for future work to explore solutions to the invariance problem.