mlsanityData to 7 Oct 2026

Random Neural Networks in the Infinite Width Limit as Gaussian Processes

arXiv 2107.01562 ↗First posted 4 Jul 2021math.PR cs.LG math.STcited by 43

Not on the map. Why?

Abstract (arXiv)

This article gives a new proof that fully connected neural networks with random weights and biases converge to Gaussian processes in the regime where the input dimension, output dimension, and depth are kept fixed, while the hidden layer widths tend to infinity. Unlike prior work, convergence is shown assuming only moment conditions for the distribution of weights and for quite general non-linearities.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 33 of its 63 references that are in the map. First posted on arXiv 4 Jul 2021.

Builds on

33 of 63 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2016Error bounds for approximations with deep ReLU networkscited by 567
  7. 2019Benign Overfitting in Linear Regressioncited by 544
  8. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  9. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  10. 2021The Principles of Deep Learning Theorycited by 169
  11. 2019Complexity of Linear Regions in Deep Networkscited by 166
  12. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
21 more
  1. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  2. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  3. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  4. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  5. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  6. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  7. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  8. 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
  9. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  10. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  11. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  12. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  13. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  14. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  15. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  16. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  17. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  18. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  19. 2019Nonlinear Approximation and (Deep) ReLU Networkscited by 45
  20. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  21. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 2; 2023: 8; 2024: 11; 2025: 15; 2026: 4.

43 papers.

Sort:

  1. 2025Practical Efficiency of Muon for Pretrainingcited by 74
  2. 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 24
  3. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  4. 2023Structures of Neural Network Effective Theoriescited by 13
  5. 2021Rate of Convergence of Polynomial Networks to Gaussian Processescited by 12
  6. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  7. 2023Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processescited by 11
  8. 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalitiescited by 10
  9. 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 10
  10. 2024Bayesian RG Flow in Neural Network Field Theoriescited by 9
  11. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  12. 2021Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applicationscited by 8
31 more
  1. 2023A Quantitative Functional Central Limit Theorem for Shallow Neural Networkscited by 8
  2. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  3. 2021Depth induces scale-averaging in overparameterized linear Bayesian neural networkscited by 7
  4. 2025Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regimecited by 6
  5. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  6. 2024On the Impacts of the Random Initialization in the Neural Tangent Kernel Theorycited by 5
  7. 2024Neural reproducing kernel Banach spaces and representer theorems for deep networkscited by 4
  8. 2023Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of trainingcited by 3
  9. 2024Spectral complexity of deep neural networkscited by 3
  10. 2024Random ReLU Neural Networks as Non-Gaussian Processescited by 3
  11. 2024Large Deviations of Gaussian Neural Networks with ReLU activationcited by 3
  12. 2024Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selectioncited by 3
  13. 2024Proportional infinite-width infinite-depth limit for deep linear neural networkscited by 3
  14. 2025Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operatorscited by 3
  15. 2025Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weightscited by 3
  16. 2025Temperature is All You Need for Generalization in Langevin Dynamics and other Markov Processescited by 2
  17. 2022Convergence of neural networks to Gaussian mixture distributioncited by 1
  18. 2025Fractal and Regular Geometry of Deep Neural Networkscited by 1
  19. 2025Thompson Sampling in Function Spaces via Neural Operatorscited by 1
  20. 2025Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networkscited by 1
  21. 2026How Controlling the Variance can Improve Training Stability of Sparsely Activated DNNs and CNNscited by 1
  22. 2023Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processescited by 0
  23. 2024Effective Non-Random Extreme Learning Machinecited by 0
  24. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 0
  25. 2025Deep Neural Nets as Hamiltonianscited by 0
  26. 2025Critical Points of Random Neural Networkscited by 0
  27. 2025Posterior Bayesian Neural Networks with Dependent Weightscited by 0
  28. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0
  29. 2026Large deviation principles for convolutional Bayesian neural networkscited by 0
  30. 2026Optimal Non-Asymptotic Edgeworth Expansions for Multivariate Neural Network Outputscited by 0
  31. 2026High Probability Derivative Bounds for Random tanh Neural Networks on a Hypercubecited by 0