# Deep Stable neural networks: large-width asymptotics and convergence rates

arXiv: https://arxiv.org/abs/2108.02316 · HTML: https://mlsanity.com/paper/2108.02316/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 2 Aug 2021.

## Builds on (most cited first, 23 of 23)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Gaussian Process Behaviour in Wide Deep Neural Networks](https://mlsanity.com/paper/1804.11271/) (2018)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes](https://mlsanity.com/paper/1810.05148/) (2018)
- [Finite Versus Infinite Neural Networks: an Empirical Study](https://mlsanity.com/paper/2007.15801/) (2020)
- [Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes](https://mlsanity.com/paper/1910.12478/) (2019)
- [Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation](https://mlsanity.com/paper/1902.04760/) (2019)
- [Deep Convolutional Networks as shallow Gaussian Processes](https://mlsanity.com/paper/1808.05587/) (2018)
- [Bayesian Neural Network Priors Revisited](https://mlsanity.com/paper/2102.06571/) (2021)
- [On the Impact of the Activation Function on Deep Neural Networks Training](https://mlsanity.com/paper/1902.06853/) (2019)
- [The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization](https://mlsanity.com/paper/2106.04013/) (2021)
- [Steps Toward Deep Kernel Methods from Infinite Neural Networks](https://mlsanity.com/paper/1508.05133/) (2015)
- [Stable behaviour of infinitely wide deep neural networks](https://mlsanity.com/paper/2003.00394/) (2020)
- [Finite size corrections for neural network Gaussian processes](https://mlsanity.com/paper/1908.10030/) (2019)
- [Non-asymptotic approximations of neural networks by Gaussian processes](https://mlsanity.com/paper/2102.08668/) (2021)
- [On the asymptotics of wide networks with polynomial activations](https://mlsanity.com/paper/2006.06687/) (2020)
- [Asymptotics of Wide Convolutional Neural Networks](https://mlsanity.com/paper/2008.08675/) (2020)
- [Scaling Properties of Deep Residual Networks](https://mlsanity.com/paper/2105.12245/) (2021)
- [Mediated Experts for Deep Convolutional Networks](https://mlsanity.com/paper/1511.06072/) (2015)

## Built on by (most cited first, 8 of 8)

- [Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalities](https://mlsanity.com/paper/2304.04010/) (2023)
- [Gaussian random field approximation via Stein's method with applications to wide random neural networks](https://mlsanity.com/paper/2306.16308/) (2023)
- [Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functions](https://mlsanity.com/paper/2304.04008/) (2023)
- [Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weights](https://mlsanity.com/paper/2507.12686/) (2025)
- [Convergence of neural networks to Gaussian mixture distribution](https://mlsanity.com/paper/2204.12100/) (2022)
- [Large-width asymptotics for ReLU neural networks with α-Stable initializations](https://mlsanity.com/paper/2206.08065/) (2022)
- [Deep Kernel Posterior Learning under Infinite Variance Prior Weights](https://mlsanity.com/paper/2410.01284/) (2024)
- [Posterior Bayesian Neural Networks with Dependent Weights](https://mlsanity.com/paper/2507.22095/) (2025)

## Abstract (arXiv)

In modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotic properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and Gaussian stochastic processes (SPs). Such an interplay has proved to be critical in Bayesian inference under Gaussian SP priors, kernel regression for infinitely wide deep NNs trained via gradient descent, and information propagation within infinitely wide NNs. Motivated by empirical analyses that show the potential of replacing Gaussian distributions with Stable distributions for the NN's weights, in this paper we present a rigorous analysis of the large-width asymptotic behaviour of (fully connected) feed-forward deep Stable NNs, i.e. deep NNs with Stable-distributed weights. We show that as the width goes to infinity jointly over the NN's layers, i.e. the ``joint growth" setting, a rescaled deep Stable NN converges weakly to a Stable SP whose distribution is characterized recursively through the NN's layers. Because of the non-triangular structure of the NN, this is a non-standard asymptotic problem, to which we propose an inductive approach of independent interest. Then, we establish sup-norm convergence rates of the rescaled deep Stable NN to the Stable SP, under the ``joint growth" and a ``sequential growth" of the width over the NN's layers. Such a result provides the difference between the ``joint growth" and the ``sequential growth" settings, showing that the former leads to a slower rate than the latter, depending on the depth of the layer and the number of inputs of the NN. Our work extends some recent results on infinitely wide limits for deep Gaussian NNs to the more general deep Stable NNs, providing the first result on convergence rates in the ``joint growth" setting.