# Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processes

arXiv: https://arxiv.org/abs/2312.11737 · HTML: https://mlsanity.com/paper/2312.11737/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 13 of 13)

- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks](https://mlsanity.com/paper/2001.11443/) (2020)
- [Random Neural Networks in the Infinite Width Limit as Gaussian Processes](https://mlsanity.com/paper/2107.01562/) (2021)
- [Exact posterior distributions of wide Bayesian neural networks](https://mlsanity.com/paper/2006.10541/) (2020)
- [Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks](https://mlsanity.com/paper/2203.07379/) (2022)
- [Quantitative CLTs in Deep Neural Networks](https://mlsanity.com/paper/2307.06092/) (2023)
- [Non-asymptotic approximations of neural networks by Gaussian processes](https://mlsanity.com/paper/2102.08668/) (2021)
- [Large-width functional asymptotics for deep Gaussian neural networks](https://mlsanity.com/paper/2102.10307/) (2021)
- [Rate of Convergence of Polynomial Networks to Gaussian Processes](https://mlsanity.com/paper/2111.03175/) (2021)
- [Gaussian random field approximation via Stein's method with applications to wide random neural networks](https://mlsanity.com/paper/2306.16308/) (2023)
- [A Quantitative Functional Central Limit Theorem for Shallow Neural Networks](https://mlsanity.com/paper/2306.16932/) (2023)

## Built on by (most cited first, 13 of 13)

- [Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layers](https://mlsanity.com/paper/2406.03260/) (2024)
- [Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weights](https://mlsanity.com/paper/2507.12686/) (2025)
- [Proportional infinite-width infinite-depth limit for deep linear neural networks](https://mlsanity.com/paper/2411.15267/) (2024)
- [Student-t processes as infinite-width limits of posterior Bayesian neural networks](https://mlsanity.com/paper/2502.04247/) (2025)
- [Universality in Deep Neural Networks: An approach via the Lindeberg exchange principle](https://mlsanity.com/paper/2605.02771/) (2026)
- [Quantitative convergence of trained single layer neural networks to Gaussian processes](https://mlsanity.com/paper/2509.24544/) (2025)
- [Bayesian Inference with Shaped Deep Non-linear MLPs](https://mlsanity.com/paper/2605.30860/) (2026)
- [Quantitative Gaussian-Process limits of Tensor Programs](https://mlsanity.com/paper/2607.06290/) (2026)
- [Posterior Bayesian Neural Networks with Dependent Weights](https://mlsanity.com/paper/2507.22095/) (2025)
- [Large deviation principles for convolutional Bayesian neural networks](https://mlsanity.com/paper/2603.06023/) (2026)
- [Optimal Non-Asymptotic Edgeworth Expansions for Multivariate Neural Network Outputs](https://mlsanity.com/paper/2605.24072/) (2026)
- [Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product](https://mlsanity.com/paper/2606.30831/) (2026)
- [Lifting Finite-Sample Gaussian Approximation of Wide Neural Networks to Functional Convergence](https://mlsanity.com/paper/2610.18166/) (2026)

## Abstract (arXiv)

We establish novel rates for the Gaussian approximation of random deep neural networks with Gaussian parameters (weights and biases) and Lipschitz activation functions, in the wide limit. Our bounds apply for the joint output of a network evaluated any finite input set, provided a certain non-degeneracy condition of the infinite-width covariances holds. We demonstrate that the distance between the network output and the corresponding Gaussian approximation scales inversely with the width of the network, exhibiting faster convergence than the naive heuristic suggested by the central limit theorem. We also apply our bounds to obtain theoretical approximations for the exact Bayesian posterior distribution of the network, when the likelihood is a bounded Lipschitz function of the network output evaluated on a (finite) training set. This includes popular cases such as the Gaussian likelihood, i.e. exponential of minus the mean squared error.