# Scale Mixtures of Neural Network Gaussian Processes

arXiv: https://arxiv.org/abs/2107.01408 · HTML: https://mlsanity.com/paper/2107.01408/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 3 Jul 2021.

## Builds on (most cited first, 13 of 13)

- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes](https://mlsanity.com/paper/1910.12478/) (2019)
- [Neural Tangents: Fast and Easy Infinite Neural Networks in Python](https://mlsanity.com/paper/1912.02803/) (2019)
- [Deep Convolutional Networks as shallow Gaussian Processes](https://mlsanity.com/paper/1808.05587/) (2018)
- [Infinite attention: NNGP and NTK for deep attention networks](https://mlsanity.com/paper/2006.10540/) (2020)
- [Student-t Processes as Alternatives to Gaussian Processes](https://mlsanity.com/paper/1402.4306/) (2014)
- [Stable behaviour of infinitely wide deep neural networks](https://mlsanity.com/paper/2003.00394/) (2020)
- [Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit](https://mlsanity.com/paper/2010.07355/) (2020)
- [Kullback-Leibler Divergence for the Normal-Gamma Distribution](https://mlsanity.com/paper/1611.01437/) (2016)
- [Infinite-channel deep stable convolutional neural networks](https://mlsanity.com/paper/2102.03739/) (2021)

## Built on by (most cited first, 5 of 5)

- [Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility](https://mlsanity.com/paper/2205.08187/) (2022)
- [Depth induces scale-averaging in overparameterized linear Bayesian neural networks](https://mlsanity.com/paper/2111.11954/) (2021)
- [Dimension Agnostic Neural Processes](https://mlsanity.com/paper/2502.20661/) (2025)
- [Variational Bayesian Pseudo-Coreset](https://mlsanity.com/paper/2502.21143/) (2025)
- [Rethinking Likelihood distributions: Student's t Likelihood Boosts Bayesian Neural Network Performance](https://mlsanity.com/paper/2607.25376/) (2026)

## Abstract (arXiv)

Recent works have revealed that infinitely-wide feed-forward or recurrent neural networks of any architecture correspond to Gaussian processes referred to as Neural Network Gaussian Processes (NNGPs). While these works have extended the class of neural networks converging to Gaussian processes significantly, however, there has been little focus on broadening the class of stochastic processes that such neural networks converge to. In this work, inspired by the scale mixture of Gaussian random variables, we propose the scale mixture of NNGPs for which we introduce a prior distribution on the scale of the last-layer parameters. We show that simply introducing a scale prior on the last-layer parameters can turn infinitely-wide neural networks of any architecture into a richer class of stochastic processes. With certain scale priors, we obtain heavy-tailed stochastic processes, and in the case of inverse gamma priors, we recover Student's t processes. We further analyze the distributions of the neural networks initialized with our prior setting and trained with gradient descents and obtain similar results as for NNGPs. We present a practical posterior-inference algorithm for the scale mixture of NNGPs and empirically demonstrate its usefulness on regression and classification tasks. In particular, we show that in both tasks, the heavy-tailed stochastic processes obtained from our framework are robust to out-of-distribution data.