# Functional Stochastic Gradient MCMC for Bayesian Neural Networks

arXiv: https://arxiv.org/abs/2409.16632 · HTML: https://mlsanity.com/paper/2409.16632/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 66 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Advances in Variational Inference](https://mlsanity.com/paper/1711.05597/) (2017)
- [Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning](https://mlsanity.com/paper/1902.03932/) (2019)
- [An Information-Theoretic Analysis of Thompson Sampling](https://mlsanity.com/paper/1403.5341/) (2014)
- [A Complete Recipe for Stochastic Gradient MCMC](https://mlsanity.com/paper/1506.04696/) (2015)
- [Functional Variational Bayesian Neural Networks](https://mlsanity.com/paper/1903.05779/) (2019)
- [Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks](https://mlsanity.com/paper/1512.07666/) (2015)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks](https://mlsanity.com/paper/1912.10481/) (2019)
- [All You Need is a Good Functional Prior for Bayesian Deep Learning](https://mlsanity.com/paper/2011.12829/) (2020)
- [On Sparse variational methods and the Kullback-Leibler divergence between stochastic processes](https://mlsanity.com/paper/1504.07027/) (2015)
- [Stochastic gradient Markov chain Monte Carlo](https://mlsanity.com/paper/1907.06986/) (2019)
- [Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting](https://mlsanity.com/paper/2010.06772/) (2020)
- [Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Tasks](https://mlsanity.com/paper/2211.12717/) (2022)
- [A Spectral Approach to Gradient Estimation for Implicit Distributions](https://mlsanity.com/paper/1806.02925/) (2018)
- [Challenges in Markov chain Monte Carlo for Bayesian neural networks](https://mlsanity.com/paper/1910.06539/) (2019)
- [Tractable Function-Space Variational Inference in Bayesian Neural Networks](https://mlsanity.com/paper/2312.17199/) (2023)
- [Understanding Variational Inference in Function-Space](https://mlsanity.com/paper/2011.09421/) (2020)
- [Avoiding pathologies in very deep networks](https://mlsanity.com/paper/1402.5836/) (2014)
- [Exact Langevin Dynamics with Stochastic Gradients](https://mlsanity.com/paper/2102.01691/) (2021)
- [Control Variates for Stochastic Gradient MCMC](https://mlsanity.com/paper/1706.05439/) (2017)
- [Generalized Variational Inference in Function Spaces: Gaussian Measures meet Bayesian Deep Learning](https://mlsanity.com/paper/2205.06342/) (2022)

## Built on by (most cited first, 2 of 2)

- [Bayesian Lottery Ticket Hypothesis](https://mlsanity.com/paper/2602.18825/) (2026)
- [On-the-Fly Fine-Tuning of Foundational Neural Network Potentials: A Bayesian Neural Network Approach](https://mlsanity.com/paper/2507.13805/) (2025)

## Abstract (arXiv)

Classical parameter-space Bayesian inference for Bayesian neural networks (BNNs) suffers from several unresolved prior issues, such as knowledge encoding intractability and pathological behaviours in deep networks, which can lead to improper posterior inference. To address these issues, functional Bayesian inference has recently been proposed leveraging functional priors, such as the emerging functional variational inference. In addition to variational methods, stochastic gradient Markov Chain Monte Carlo (MCMC) is another scalable and effective inference method for BNNs to asymptotically generate samples from the true posterior by simulating continuous dynamics. However, existing MCMC methods perform solely in parameter space and inherit the unresolved prior issues, while extending these dynamics to function space is a non-trivial undertaking. In this paper, we introduce novel functional MCMC schemes, including stochastic gradient versions, based on newly designed diffusion dynamics that can incorporate more informative functional priors. Moreover, we prove that the stationary measure of these functional dynamics is the target posterior over functions. Our functional MCMC schemes demonstrate improved performance in both predictive accuracy and uncertainty quantification on several tasks compared to naive parameter-space MCMC and functional variational inference.