mlsanityData to 7 Oct 2026

Non-asymptotic approximations of neural networks by Gaussian processes

arXiv 2102.08668 ↗First posted 17 Feb 2021math.PR cs.LG stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

We study the extent to which wide neural networks may be approximated by Gaussian processes when initialized with random weights. It is a well-established fact that as the width of a network goes to infinity, its law converges to that of a Gaussian process. We make this quantitative by establishing explicit convergence rates for the central limit theorem in an infinite-dimensional functional space, metrized with a natural transportation distance. We identify two regimes of interest; when the activation function is polynomial, its degree determines the rate of convergence, while for non-polynomial activations, the rate is governed by the smoothness of the function.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 17 Feb 2021.

Builds on

24 of 38 references.

References · 24

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018On Lazy Training in Differentiable Programmingcited by 671
  3. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  5. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  6. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  7. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  8. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
  9. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  10. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  11. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 137
  12. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
12 more
  1. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 89
  2. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  3. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  4. 2019Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginalscited by 32
  5. 2015Steps Toward Deep Kernel Methods from Infinite Neural Networkscited by 28
  6. 2014Testing for high-dimensional geometry in random graphscited by 25
  7. 2019Finite size corrections for neural network Gaussian processescited by 22
  8. 2020The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networkscited by 20
  9. 2020On the asymptotics of wide networks with polynomial activationscited by 17
  10. 2020Asymptotics of Wide Convolutional Neural Networkscited by 15
  11. 2019Effect of Activation Functions on the Training of Overparametrized Neural Netscited by 14
  12. 2020Function approximation by neural nets in the mean-field regime: Entropic regularization and controlled McKean-Vlasov dynamicscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 2; 2023: 7; 2024: 2; 2025: 2; 2026: 1.

18 papers.

Sort:

  1. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 43
  2. 2023Generalization Ability of Wide Neural Networks on ℝcited by 28
  3. 2021A single gradient step finds adversarial examples on random two-layers neural networkscited by 26
  4. 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 24
  5. 2023Quantitative CLTs in Deep Neural Networkscited by 19
  6. 2021Rate of Convergence of Polynomial Networks to Gaussian Processescited by 12
  7. 2023Fundamental limits of overparametrized shallow neural networks for supervised learningcited by 12
  8. 2023Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processescited by 11
  9. 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalitiescited by 10
  10. 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 10
  11. 2021Deep Stable neural networks: large-width asymptotics and convergence ratescited by 8
  12. 2023A Quantitative Functional Central Limit Theorem for Shallow Neural Networkscited by 8
6 more
  1. 2024Variational Search Distributionscited by 6
  2. 2024Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selectioncited by 3
  3. 2025Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weightscited by 3
  4. 2022Large-width asymptotics for ReLU neural networks with α-Stable initializationscited by 0
  5. 2025Posterior Bayesian Neural Networks with Dependent Weightscited by 0
  6. 2026Quantitative Gaussian-Process limits of Tensor Programscited by 0