# Challenges in Markov chain Monte Carlo for Bayesian neural networks

arXiv: https://arxiv.org/abs/1910.06539 · HTML: https://mlsanity.com/paper/1910.06539/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2019.

## Built on by (most cited first, 30 of 44)

- [Variational Bayesian Last Layers](https://mlsanity.com/paper/2404.11599/) (2024)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetry](https://mlsanity.com/paper/2304.02902/) (2023)
- [The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof](https://mlsanity.com/paper/2405.20231/) (2024)
- [A Comprehensive Review of Digital Twin -- Part 2: Roles of Uncertainty Quantification and Optimization, a Battery Digital Twin, and Perspectives](https://mlsanity.com/paper/2208.12904/) (2022)
- [Accelerated Parallel Tempering via Neural Transports](https://mlsanity.com/paper/2502.10328/) (2025)
- [How Interpretable Are Interpretable Graph Neural Networks?](https://mlsanity.com/paper/2406.07955/) (2024)
- [Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics](https://mlsanity.com/paper/2410.19780/) (2024)
- [Variational Bayes Deep Operator Network: A data-driven Bayesian solver for parametric differential equations](https://mlsanity.com/paper/2206.05655/) (2022)
- [Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks](https://mlsanity.com/paper/2502.06335/) (2025)
- [Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics](https://mlsanity.com/paper/2303.05101/) (2023)
- [Uncertainty-Aware Mixed-Variable Machine Learning for Materials Design](https://mlsanity.com/paper/2207.04994/) (2022)
- [Efficient Bayesian Physics Informed Neural Networks for Inverse Problems via Ensemble Kalman Inversion](https://mlsanity.com/paper/2303.07392/) (2023)
- [Neural Mixed Effects for Nonlinear Personalized Predictions](https://mlsanity.com/paper/2306.08149/) (2023)
- [Learning to solve Bayesian inverse problems: An amortized variational inference approach using Gaussian and Flow guides](https://mlsanity.com/paper/2305.20004/) (2023)
- [Bayesian neural networks for predicting uncertainty in full-field material response](https://mlsanity.com/paper/2406.14838/) (2024)
- [Randomized prior wavelet neural operator for uncertainty quantification](https://mlsanity.com/paper/2302.01051/) (2023)
- [On Feynman--Kac training of partial Bayesian neural networks](https://mlsanity.com/paper/2310.19608/) (2023)
- [Richer Bayesian Last Layers with Subsampled NTK Features](https://mlsanity.com/paper/2602.01279/) (2026)
- [Approximate blocked Gibbs sampling for Bayesian neural networks](https://mlsanity.com/paper/2208.11389/) (2022)
- [The Deep Arbitrary Polynomial Chaos Neural Network or how Deep Artificial Neural Networks could benefit from Data-Driven Homogeneous Chaos Theory](https://mlsanity.com/paper/2306.14753/) (2023)
- [On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2603.22030/) (2026)
- [Functional Stochastic Gradient MCMC for Bayesian Neural Networks](https://mlsanity.com/paper/2409.16632/) (2024)
- [Scalable Bayesian Physics-Informed Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2501.08501/) (2025)
- [Gated Linear Model induced U-net for surrogate modeling and uncertainty quantification](https://mlsanity.com/paper/2111.05123/) (2021)
- [An Ensemble-Based Deep Framework for Estimating Thermo-Chemical State Variables from Flamelet Generated Manifolds](https://mlsanity.com/paper/2211.14098/) (2022)
- [Introduction and Exemplars of Uncertainty Decomposition](https://mlsanity.com/paper/2211.15475/) (2022)
- [Robust Deep Reinforcement Learning for Inverter-based Volt-Var Control in Partially Observable Distribution Networks](https://mlsanity.com/paper/2408.06776/) (2024)
- [Bayesian Computation in Deep Learning](https://mlsanity.com/paper/2502.18300/) (2025)
- [Understanding the Trade-offs in Accuracy and Uncertainty Quantification: Architecture and Inference Choices in Bayesian Neural Networks](https://mlsanity.com/paper/2503.11808/) (2025)

## Abstract (arXiv)

Markov chain Monte Carlo (MCMC) methods have not been broadly adopted in Bayesian neural networks (BNNs). This paper initially reviews the main challenges in sampling from the parameter posterior of a neural network via MCMC. Such challenges culminate to lack of convergence to the parameter posterior. Nevertheless, this paper shows that a non-converged Markov chain, generated via MCMC sampling from the parameter space of a neural network, can yield via Bayesian marginalization a valuable posterior predictive distribution of the output of the neural network. Classification examples based on multilayer perceptrons showcase highly accurate posterior predictive distributions. The postulate of limited scope for MCMC developments in BNNs is partially valid; an asymptotically exact parameter posterior seems less plausible, yet an accurate posterior predictive distribution is a tenable research avenue.