# Richer priors for infinitely wide multi-layer perceptrons

arXiv: https://arxiv.org/abs/1911.12927 · HTML: https://mlsanity.com/paper/1911.12927/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2019.

## Built on by (most cited first, 9 of 9)

- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [When does compositional structure yield compositional generalization? A kernel theory](https://mlsanity.com/paper/2405.16391/) (2024)
- [Fast Neural Kernel Embeddings for General Activations](https://mlsanity.com/paper/2209.04121/) (2022)
- [Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility](https://mlsanity.com/paper/2205.08187/) (2022)
- [Squared Neural Families: A New Class of Tractable Density Models](https://mlsanity.com/paper/2305.13552/) (2023)
- [Revisiting the Equivalence of Bayesian Neural Networks and Gaussian Processes: On the Importance of Learning Activations](https://mlsanity.com/paper/2410.15777/) (2024)
- [Deep Kernel Posterior Learning under Infinite Variance Prior Weights](https://mlsanity.com/paper/2410.01284/) (2024)
- [Hi-fi functional priors by learning activations](https://mlsanity.com/paper/2508.08880/) (2025)

## Abstract (arXiv)

It is well-known that the distribution over functions induced through a zero-mean iid prior distribution over the parameters of a multi-layer perceptron (MLP) converges to a Gaussian process (GP), under mild conditions. We extend this result firstly to independent priors with general zero or non-zero means, and secondly to a family of partially exchangeable priors which generalise iid priors. We discuss how the second prior arises naturally when considering an equivalence class of functions in an MLP and through training processes such as stochastic gradient descent. The model resulting from partially exchangeable priors is a GP, with an additional level of inference in the sense that the prior and posterior predictive distributions require marginalisation over hyperparameters. We derive the kernels of the limiting GP in deep MLPs, and show empirically that these kernels avoid certain pathologies present in previously studied priors. We empirically evaluate our claims of convergence by measuring the maximum mean discrepancy between finite width models and limiting models. We compare the performance of our new limiting model to some previously discussed models on synthetic regression problems. We observe increasing ill-conditioning of the marginal likelihood and hyper-posterior as the depth of the model increases, drawing parallels with finite width networks which require notoriously involved optimisation tricks.