# Neural Network Gaussian Processes by Increasing Depth

arXiv: https://arxiv.org/abs/2108.12862 · HTML: https://mlsanity.com/paper/2108.12862/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 29 Aug 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes](https://mlsanity.com/paper/1910.12478/) (2019)
- [Deep Convolutional Networks as shallow Gaussian Processes](https://mlsanity.com/paper/1808.05587/) (2018)
- [Infinite attention: NNGP and NTK for deep attention networks](https://mlsanity.com/paper/2006.10540/) (2020)
- [Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks](https://mlsanity.com/paper/2012.11654/) (2020)
- [CT Super-resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble(GAN-CIRCLE)](https://mlsanity.com/paper/1808.04256/) (2018)
- [Towards NNGP-guided Neural Architecture Search](https://mlsanity.com/paper/2011.06006/) (2020)
- [Neural-net-induced Gaussian process regression for function approximation and PDE solution](https://mlsanity.com/paper/1806.11187/) (2018)
- [Bayesian Image Classification with Deep Convolutional Gaussian Processes](https://mlsanity.com/paper/1902.05888/) (2019)
- [On a Sparse Shortcut Topology of Artificial Neural Networks](https://mlsanity.com/paper/1811.09003/) (2018)
- [Large-width functional asymptotics for deep Gaussian neural networks](https://mlsanity.com/paper/2102.10307/) (2021)
- [Interpretable deep Gaussian processes with moments](https://mlsanity.com/paper/1905.10963/) (2019)
- [Quasi-Equivalence of Width and Depth of Neural Networks](https://mlsanity.com/paper/2002.02515/) (2020)

## Built on by (most cited first, 4 of 4)

- [BGDB: Bernoulli-Gaussian Decision Block with Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2409.13116/) (2024)
- [A Unified Kernel for Neural Network Learning](https://mlsanity.com/paper/2403.17467/) (2024)
- [Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernels](https://mlsanity.com/paper/2511.05585/) (2025)
- [On the Approximation and Complexity of Deep Neural Networks to Invariant Functions](https://mlsanity.com/paper/2210.15279/) (2022)

## Abstract (arXiv)

Recent years have witnessed an increasing interest in the correspondence between infinitely wide networks and Gaussian processes. Despite the effectiveness and elegance of the current neural network Gaussian process theory, to the best of our knowledge, all the neural network Gaussian processes are essentially induced by increasing width. However, in the era of deep learning, what concerns us more regarding a neural network is its depth as well as how depth impacts the behaviors of a network. Inspired by a width-depth symmetry consideration, we use a shortcut network to show that increasing the depth of a neural network can also give rise to a Gaussian process, which is a valuable addition to the existing theory and contributes to revealing the true picture of deep learning. Beyond the proposed Gaussian process by depth, we theoretically characterize its uniform tightness property and the smallest eigenvalue of the Gaussian process kernel. These characterizations can not only enhance our understanding of the proposed depth-induced Gaussian process but also pave the way for future applications. Lastly, we examine the performance of the proposed Gaussian process by regression experiments on two benchmark data sets.