mlsanityData to 7 Oct 2026

Stable behaviour of infinitely wide deep neural networks

arXiv 2003.00394 ↗First posted 1 Mar 2020stat.ML cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimensional distributions are multivariate stable distributions. The limiting process is referred to as the stable process, and it generalizes the class of Gaussian processes recently obtained as infinite wide limits of NNs (Matthews at al., 2018b). Parameters of the stable process can be computed via an explicit recursion over the layers of the network. Our result contributes to the theory of fully connected feed-forward deep NNs, and it paves the way to expand recent lines of research that rely on Gaussian infinite wide limits.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 5; 2023: 4; 2024: 3; 2025: 1; 2026: 2.

23 papers.

Sort:

  1. 2021Bayesian Neural Network Priors Revisitedcited by 121
  2. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  3. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  4. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  5. 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 24
  6. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  7. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  8. 2022Feature Space Particle Inference for Neural Network Ensemblescited by 9
  9. 2021Deep Stable neural networks: large-width asymptotics and convergence ratescited by 8
  10. 2021α-Stable convergence of heavy-tailed infinitely-wide neural networkscited by 5
  11. 2021Scale Mixtures of Neural Network Gaussian Processescited by 5
  12. 2021Bayesian neural network unit priors and generalized Weibull-tail propertycited by 5
11 more
  1. 2023Infinitely wide limits for deep Stable neural networks: sub-linear, linear and super-linear activation functionscited by 5
  2. 2021Dependence between Bayesian neural network unitscited by 3
  3. 2023Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learningcited by 3
  4. 2023Posterior Inference on Shallow Infinitely Wide Bayesian Neural Networks under Weights with Unbounded Variancecited by 3
  5. 2025Sharp High-Probability Rates for Nonlinear SGD under Heavy-Tailed Noise via Symmetrizationcited by 3
  6. 2024Nonlinear Stochastic Gradient Descent and Heavy-tailed Noise: A Unified Framework and High-probability Guaranteescited by 1
  7. 2022Large-width asymptotics for ReLU neural networks with α-Stable initializationscited by 0
  8. 2024Deep Kernel Posterior Learning under Infinite Variance Prior Weightscited by 0
  9. 2024Variational Bayesian Bow tie Neural Networks with Shrinkagecited by 0
  10. 2026Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient samplingcited by 0
  11. 2026How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPscited by 0