# Trace-class Gaussian priors for Bayesian learning of neural networks with MCMC

arXiv: https://arxiv.org/abs/2012.10943 · HTML: https://mlsanity.com/paper/2012.10943/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2020.

## Built on by (most cited first, 5 of 5)

- [Posterior and variational inference for deep neural networks with heavy-tailed weights](https://mlsanity.com/paper/2406.03369/) (2024)
- [Function-Space MCMC for Bayesian Wide Neural Networks](https://mlsanity.com/paper/2408.14325/) (2024)
- [Variational Bayesian Bow tie Neural Networks with Shrinkage](https://mlsanity.com/paper/2411.11132/) (2024)
- [Partial Trace-Class Bayesian Neural Networks](https://mlsanity.com/paper/2511.01628/) (2025)
- [Bayesian learning for the stochastic shortest path problem](https://mlsanity.com/paper/2606.04845/) (2026)

## Abstract (arXiv)

This paper introduces a new neural network based prior for real valued functions on mathbb Rd which, by construction, is more easily and cheaply scaled up in the domain dimension d compared to the usual Karhunen-Lo`eve function space prior. The new prior is a Gaussian neural network prior, where each weight and bias has an independent Gaussian prior, but with the key difference that the variances decrease in the width of the network in such a way that the resulting function is almost surely well defined in the limit of an infinite width network. We show that in a Bayesian treatment of inferring unknown functions, the induced posterior over functions is amenable to Monte Carlo sampling using Hilbert space Markov chain Monte Carlo (MCMC) methods. This type of MCMC is popular, e.g. in the Bayesian Inverse Problems literature, because it is stable under mesh refinement, i.e. the acceptance probability does not shrink to 0 as more parameters of the function's prior are introduced, even ad infinitum. In numerical examples we demonstrate these stated competitive advantages over other function space priors. We also implement examples in Bayesian Reinforcement Learning to automate tasks from data and demonstrate, for the first time, stability of MCMC to mesh refinement for these type of problems.