# Theoretical and Practical Advances in Hierarchical Bayesian Deep Learning Methodology

arXiv: https://arxiv.org/abs/2610.19074 · HTML: https://mlsanity.com/paper/2610.19074/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 36 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 12 of 12)

- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Hamiltonian Monte Carlo for Hierarchical Models](https://mlsanity.com/paper/1312.0906/) (2013)
- [Challenges in Markov chain Monte Carlo for Bayesian neural networks](https://mlsanity.com/paper/1910.06539/) (2019)
- [Wide Neural Networks with Bottlenecks are Deep Gaussian Processes](https://mlsanity.com/paper/2001.00921/) (2020)
- [Multilevel Monte Carlo Variational Inference](https://mlsanity.com/paper/1902.00468/) (2019)
- [Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics](https://mlsanity.com/paper/2410.19780/) (2024)
- [A Multi-level procedure for enhancing accuracy of machine learning algorithms](https://mlsanity.com/paper/1909.09448/) (2019)
- [Unbiased Kinetic Langevin Monte Carlo with Inexact Gradients](https://mlsanity.com/paper/2311.05025/) (2023)
- [Bayesian Deep Learning with Multilevel Trace-class Neural Networks](https://mlsanity.com/paper/2203.12961/) (2022)
- [Unbiased inference for discretely observed hidden Markov model diffusions](https://mlsanity.com/paper/1807.10259/) (2018)
- [Scalable Bayesian Monte Carlo: fast uncertainty estimation beyond deep ensembles](https://mlsanity.com/paper/2505.13585/) (2025)

## Abstract (arXiv)

Reliable uncertainty quantification remains an important challenge in AI. Hierarchical Bayesian methods provide a principled framework for this by representing unknown quantities through posterior distributions. Joint inference of neural-network parameters and prior hyperparameters is challenging for conventional Markov chain Monte Carlo (MCMC) methods due to strong dependencies. In this work, we introduce the multilevel particle marginal Metropolis-Hastings (MLPMMH) method, which combines pseudo-marginal inference for the hyperparameters with coupled sequential Monte Carlo (SMC) across hierarchy of network-width truncations for conditional parameter inference. The hierarchy is induced by a trace-class neural network (TNN) prior with width-dependent decay, motivated empirically by the compressible structure observed in trained neural-network parameters. Under standard multilevel coupling assumptions, we establish that MLPMMH achieves mean-square error (MSE) O(varepsilon2) at expected computational cost O(varepsilon-2), and verify the predicted rate on a Bayesian regression problem. To address larger-scale applications, we further introduce S-MLPMMH, a practical approximation that replaces the exact sampling components with a PCA-based hierarchy and scalable Bayesian Monte Carlo. Experiments on MNIST demonstrate competitive predictive performance and uncertainty quantification while exploiting the parallel structure of the approximation. Software has been made publicly available at https://github.com/submission-code-anon/mlpmmh-anon.