mlsanityData to 14 Oct 2026

Deep Stable neural networks: large-width asymptotics and convergence rates

arXiv 2108.02316 ↗First posted 2 Aug 2021cs.LG math.PR math.STcited by 8

Not on the map. Why?

Abstract (arXiv)

In modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotic properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and Gaussian stochastic processes (SPs). Such an interplay has proved to be critical in Bayesian inference under Gaussian SP priors, kernel regression for infinitely wide deep NNs trained via gradient descent, and information propagation within infinitely wide NNs. Motivated by empirical analyses that show the potential of replacing Gaussian distributions with Stable distributions for the NN's weights, in this paper we present a rigorous analysis of the large-width asymptotic behaviour of (fully connected) feed-forward deep Stable NNs, i.e. deep NNs with Stable-distributed weights. We show that as the width goes to infinity jointly over the NN's layers, i.e. the ``joint growth" setting, a rescaled deep Stable NN converges weakly to a Stable SP whose distribution is characterized recursively through the NN's layers. Because of the non-triangular structure of the NN, this is a non-standard asymptotic problem, to which we propose an inductive approach of independent interest. Then, we establish sup-norm convergence rates of the rescaled deep Stable NN to the Stable SP, under the ``joint growth" and a ``sequential growth" of the width over the NN's layers. Such a result provides the difference between the ``joint growth" and the ``sequential growth" settings, showing that the former leads to a slower rate than the latter, depending on the depth of the layer and the number of inputs of the NN. Our work extends some recent results on infinitely wide limits for deep Gaussian NNs to the more general deep Stable NNs, providing the first result on convergence rates in the ``joint growth" setting.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 23 of its 47 references that are in the map. First posted on arXiv 2 Aug 2021.

Builds on

23 of 47 references.

References · 23

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  6. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  7. 2016Deep Information Propagationcited by 195
  8. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  9. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  10. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  11. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  12. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
11 more
  1. 2021Bayesian Neural Network Priors Revisitedcited by 121
  2. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  3. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  4. 2015Steps Toward Deep Kernel Methods from Infinite Neural Networkscited by 28
  5. 2020Stable behaviour of infinitely wide deep neural networkscited by 23
  6. 2019Finite size corrections for neural network Gaussian processescited by 22
  7. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 19
  8. 2020On the asymptotics of wide networks with polynomial activationscited by 18
  9. 2020Asymptotics of Wide Convolutional Neural Networkscited by 16
  10. 2021Scaling Properties of Deep Residual Networkscited by 16
  11. 2015Mediated Experts for Deep Convolutional Networkscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 1; 2025: 2.

8 papers.

  1. 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalitiescited by 11
  2. 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 11
  3. 2023Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functionscited by 5
  4. 2025Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weightscited by 5
  5. 2022Convergence of neural networks to Gaussian mixture distributioncited by 1
  6. 2022Large-width asymptotics for ReLU neural networks with α-Stable initializationscited by 0
  7. 2024Deep Kernel Posterior Learning under Infinite Variance Prior Weightscited by 0
  8. 2025Posterior Bayesian Neural Networks with Dependent Weightscited by 0