mlsanityData to 7 Oct 2026

The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization

arXiv 2206.02768 ↗First posted 6 Jun 2022stat.ML cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix defined by the penultimate layer. In this work, we study the distribution of this random matrix. Recent work has shown that shaping the activation function as network depth grows large is necessary for this covariance matrix to be non-degenerate. However, the current infinite-width-style understanding of this shaping method is unsatisfactory for large depth: infinite-width analyses ignore the microscopic fluctuations from layer to layer, but these fluctuations accumulate over many layers. To overcome this shortcoming, we study the random covariance matrix in the shaped infinite-depth-and-width limit. We identify the precise scaling of the activation function necessary to arrive at a non-trivial limit, and show that the random covariance matrix is governed by a stochastic differential equation (SDE) that we call the Neural Covariance SDE. Using simulations, we show that the SDE closely matches the distribution of the random covariance matrix of finite networks. Additionally, we recover an if-and-only-if condition for exploding and vanishing norms of large shaped networks based on the activation function.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 37 of its 73 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

37 of 73 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  6. 2018On Lazy Training in Differentiable Programmingcited by 671
  7. 2017Deep Neural Networks as Gaussian Processescited by 617
  8. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  9. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  10. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  11. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  12. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
25 more
  1. 2021Deep learning: a statistical viewpointcited by 239
  2. 2016Deep Information Propagationcited by 194
  3. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  4. 2021The Principles of Deep Learning Theorycited by 169
  5. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
  6. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  7. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  8. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
  9. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  10. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  11. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  12. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  13. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  14. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  15. 2019Non-Gaussian processes and neural networks at finite widthscited by 78
  16. 2017Automated Scalable Bayesian Inference via Hilbert Coresetscited by 55
  17. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
  18. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  19. 2020Stable ResNetcited by 49
  20. 2020Deep Networks and the Multiple Manifold Problemcited by 35
  21. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  22. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  23. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  24. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  25. 2021Activation function design for deep networks: linearity and effective initialisationcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 12; 2025: 7; 2026: 5.

41 papers.

Sort:

By area:

  1. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  2. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  3. 2024Scalable Optimization in the Modular Normcited by 64
  4. 2025The Curse of Depth in Large Language Modelscited by 58
  5. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  6. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  7. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
  8. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  9. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  10. 2023Simplifying Transformer Blockscited by 37
  11. 2024Linguistic Collapse: Neural Collapse in (Large) Language Modelscited by 36
  12. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
29 more
  1. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  2. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  3. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  4. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  5. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  6. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  7. 2022Meta-Principled Family of Hyperparameter Scaling Strategiescited by 18
  8. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13
  9. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  10. 2025Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformercited by 12
  11. 2023Gaussian random field approximation via Stein's method with applications to wide random neural networkscited by 10
  12. 2024Understanding and Minimising Outlier Features in Neural Network Trainingcited by 10
  13. 2024Masked Completion via Structured Diffusion with White-Box Transformerscited by 9
  14. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  15. 2024Bayesian Inference with Deep Weakly Nonlinear Networkscited by 5
  16. 2023Differential Equation Scaling Limits of Shaped and Unshaped Neural Networkscited by 4
  17. 2023Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initializationcited by 3
  18. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  19. 2023Bayesian inference with finitely wide neural networkscited by 1
  20. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  21. 2024Emergence of Globally Attracting Fixed Points in Deep Neural Networks With Nonlinear Activationscited by 1
  22. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  23. 2024A Generalization Bound for Nearly-Linear Networkscited by 0
  24. 2025Deep Neural Nets as Hamiltonianscited by 0
  25. 2025Latent Zoning Network: A Unified Principle for Generative Modeling, Representation Learning, and Classificationcited by 0
  26. 2025Feature Learning Dynamics in Infinite-Depth Neural Networkscited by 0
  27. 2026Large deviation principles for convolutional Bayesian neural networkscited by 0
  28. 2026Collective Kernel EFT for Pre-activation ResNetscited by 0
  29. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0