mlsanityData to 14 Oct 2026

Quantitative CLTs in Deep Neural Networks

arXiv 2307.06092 ↗First posted 12 Jul 2023cs.LG cs.AI math.PRcited by 20

Not on the map. Why?

Abstract (arXiv)

We study the distribution of a fully connected neural network with random Gaussian weights and biases in which the hidden layer widths are proportional to a large constant n. Under mild assumptions on the non-linearity, we obtain quantitative bounds on normal approximations valid at large but finite n and any fixed network depth. Our theorems show both for the finite-dimensional distributions and the entire process, that the distance between a random fully connected network (and its derivatives) to the corresponding infinite width Gaussian process scales like n-γ for γ>0, with the exponent depending on the metric used to measure discrepancy. Our bounds are strictly stronger in terms of their dependence on network width than any previously available in the literature; in the one-dimensional case, we also prove that they are optimal, i.e., we establish matching lower bounds.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 53 of its 116 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

53 of 116 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  6. 2018On Lazy Training in Differentiable Programmingcited by 675
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  8. 2017Deep Neural Networks as Gaussian Processescited by 622
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  11. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  12. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
38 more
  1. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  2. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  3. 2021Deep learning: a statistical viewpointcited by 240
  4. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 198
  5. 2016Deep Information Propagationcited by 195
  6. 2021The Principles of Deep Learning Theorycited by 173
  7. 2019Complexity of Linear Regions in Deep Networkscited by 167
  8. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  9. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  10. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  11. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
  12. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  13. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  14. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  15. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  16. 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
  17. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  18. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  19. 2020Deep Equals Shallow for ReLU Networks in Kernel Regimescited by 85
  20. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  21. 2018The Emergence of Spectral Universality in Deep Networkscited by 74
  22. 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
  23. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  24. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  25. 2021Random Neural Networks in the Infinite Width Limit as Gaussian Processescited by 44
  26. 2019Mean Field Analysis of Deep Neural Networkscited by 42
  27. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  28. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  29. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  30. 2022Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networkscited by 25
  31. 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
  32. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  33. 2018Procrustes Metrics on Covariance Operators and Optimal Transportation of Gaussian Processescited by 19
  34. 2021Non-asymptotic approximations of neural networks by Gaussian processescited by 19
  35. 2021Large-width functional asymptotics for deep Gaussian neural networkscited by 15
  36. 2019Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefitscited by 13
  37. 2021Rate of Convergence of Polynomial Networks to Gaussian Processescited by 13
  38. 2023Non-asymptotic approximations of Gaussian neural networks via second-order Poincar'e inequalitiescited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 6; 2025: 6; 2026: 6.

20 papers.

Sort:

  1. 2023Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networkscited by 19
  2. 2023Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processescited by 13
  3. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  4. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  5. 2025Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weightscited by 5
  6. 2024Spectral complexity of deep neural networkscited by 3
  7. 2024Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selectioncited by 3
  8. 2024Proportional infinite-width infinite-depth limit for deep linear neural networkscited by 3
  9. 2025Student-t processes as infinite-width limits of posterior Bayesian neural networkscited by 3
  10. 2025Fractal and Regular Geometry of Deep Neural Networkscited by 1
  11. 2025Quantitative convergence of trained single layer neural networks to Gaussian processescited by 1
  12. 2026Quantitative Gaussian-Process limits of Tensor Programscited by 1
8 more
  1. 2024Effective Non-Random Extreme Learning Machinecited by 0
  2. 2025Critical Points of Random Neural Networkscited by 0
  3. 2025Posterior Bayesian Neural Networks with Dependent Weightscited by 0
  4. 2026Large deviation principles for convolutional Bayesian neural networkscited by 0
  5. 2026Optimal Non-Asymptotic Edgeworth Expansions for Multivariate Neural Network Outputscited by 0
  6. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0
  7. 2026High Probability Derivative Bounds for Random tanh Neural Networks on a Hypercubecited by 0
  8. 2026Lifting Finite-Sample Gaussian Approximation of Wide Neural Networks to Functional Convergencecited by 0