# Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry

arXiv: https://arxiv.org/abs/2304.02902 · HTML: https://mlsanity.com/paper/2304.02902/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 6 Apr 2023.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Essentially No Barriers in Neural Network Energy Landscape](https://mlsanity.com/paper/1803.00885/) (2018)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Subspace Inference for Bayesian Deep Learning](https://mlsanity.com/paper/1907.07504/) (2019)
- [Uncertainty in Neural Networks: Approximately Bayesian Ensembling](https://mlsanity.com/paper/1810.05546/) (2018)
- [Optimizing Mode Connectivity via Neuron Alignment](https://mlsanity.com/paper/2009.02439/) (2020)
- [Understanding predictive information criteria for Bayesian models](https://mlsanity.com/paper/1307.5928/) (2013)
- [Challenges in Markov chain Monte Carlo for Bayesian neural networks](https://mlsanity.com/paper/1910.06539/) (2019)
- [Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry](https://mlsanity.com/paper/2202.03038/) (2022)
- [Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning](https://mlsanity.com/paper/2111.03577/) (2021)
- [Parameter identifiability of a deep feedforward ReLU neural network](https://mlsanity.com/paper/2112.12982/) (2021)
- [Affine symmetries and neural network identifiability](https://mlsanity.com/paper/2006.11727/) (2020)
- [Nested hat R: Assessing the convergence of Markov chain Monte Carlo when running many short chains](https://mlsanity.com/paper/2110.13017/) (2021)
- [A Classification of G-invariant Shallow Neural Networks](https://mlsanity.com/paper/2205.09219/) (2022)
- [Bayesian neural networks and dimensionality reduction](https://mlsanity.com/paper/2008.08044/) (2020)
- [Markov Chains on Orbits of Permutation Groups](https://mlsanity.com/paper/1408.2052/) (2014)
- [Symmetry-Aware Marginal Density Estimation](https://mlsanity.com/paper/1304.2694/) (2013)

## Built on by (most cited first, 21 of 21)

- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models](https://mlsanity.com/paper/2406.11675/) (2024)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2310.08287/) (2023)
- [Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks](https://mlsanity.com/paper/2502.06335/) (2025)
- [A Compact Representation for Bayesian Neural Networks By Removing Permutation Symmetry](https://mlsanity.com/paper/2401.00611/) (2023)
- [On permutation symmetries in Bayesian neural network posteriors: a variational perspective](https://mlsanity.com/paper/2310.10171/) (2023)
- [Assessment of Uncertainty Quantification in Universal Differential Equations](https://mlsanity.com/paper/2406.08853/) (2024)
- [On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2603.22030/) (2026)
- [On the Epistemic Uncertainty of Overparametrized Neural Networks](https://mlsanity.com/paper/2605.25234/) (2026)
- [The Empirical Impact of Reducing Symmetries on the Performance of Deep Ensembles and MoE](https://mlsanity.com/paper/2502.17391/) (2025)
- [Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning](https://mlsanity.com/paper/2605.21765/) (2026)
- [Structured Partial Stochasticity in Bayesian Neural Networks](https://mlsanity.com/paper/2405.17666/) (2024)
- [On Uniform, Bayesian, and PAC-Bayesian Deep Ensembles](https://mlsanity.com/paper/2406.05469/) (2024)
- [Exchangeability in Neural Network and its Application to Dynamic Pruning](https://mlsanity.com/paper/2506.02210/) (2025)
- [Position: There Is No Free Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2506.03670/) (2025)
- [Constructive Disintegration and Conditional Modes](https://mlsanity.com/paper/2508.00617/) (2025)
- [Self-Supervised Laplace Approximation for Bayesian Uncertainty Quantification](https://mlsanity.com/paper/2605.12208/) (2026)
- [Discovering Symmetries in Neural Network Parameter Spaces](https://mlsanity.com/paper/2609.33527/) (2026)
- [How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?](https://mlsanity.com/paper/2610.13764/) (2026)

## Abstract (arXiv)

Bayesian inference in deep neural networks is challenging due to the high-dimensional, strongly multi-modal parameter posterior density landscape. Markov chain Monte Carlo approaches asymptotically recover the true posterior but are considered prohibitively expensive for large modern architectures. Local methods, which have emerged as a popular alternative, focus on specific parameter regions that can be approximated by functions with tractable integrals. While these often yield satisfactory empirical results, they fail, by definition, to account for the multi-modality of the parameter posterior. In this work, we argue that the dilemma between exact-but-unaffordable and cheap-but-inexact approaches can be mitigated by exploiting symmetries in the posterior landscape. Such symmetries, induced by neuron interchangeability and certain activation functions, manifest in different parameter values leading to the same functional output value. We show theoretically that the posterior predictive density in Bayesian neural networks can be restricted to a symmetry-free parameter reference set. By further deriving an upper bound on the number of Monte Carlo chains required to capture the functional diversity, we propose a straightforward approach for feasible Bayesian inference. Our experiments suggest that efficient sampling is indeed possible, opening up a promising path to accurate uncertainty quantification in deep learning.