mlsanityData to 14 Oct 2026

α-Stable convergence of heavy-tailed infinitely-wide neural networks

arXiv 2106.11064 ↗First posted 18 Jun 2021stat.ML cs.LG math.PRcited by 5

Not on the map. Why?

Abstract (arXiv)

We consider infinitely-wide multi-layer perceptrons (MLPs) which are limits of standard deep feed-forward neural networks. We assume that, for each layer, the weights of an MLP are initialized with i.i.d. samples from either a light-tailed (finite variance) or heavy-tailed distribution in the domain of attraction of a symmetric α-stable distribution, where α∈(0,2] may depend on the layer. For the bias terms of the layer, we assume i.i.d. initializations with a symmetric α-stable distribution having the same α parameter of that layer. We then extend a recent result of Favaro, Fortini, and Peluchetti (2020), to show that the vector of pre-activation values at all nodes of a given hidden layer converges in the limit, under a suitable scaling, to a vector of i.i.d. random variables with symmetric α-stable distributions.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 8 of its 25 references that are in the map. First posted on arXiv 18 Jun 2021.

Builds on

8 of 25 references.

References · 8

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  6. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  7. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  8. 2020Stable behaviour of infinitely wide deep neural networkscited by 23

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 0; 2025: 2.

5 papers.

  1. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  2. 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 11
  3. 2025To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentrationcited by 5
  4. 2023Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learningcited by 3
  5. 2025Posterior Bayesian Neural Networks with Dependent Weightscited by 0