mlsanityData to 14 Oct 2026

Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks

arXiv 2310.12079 ↗First posted 18 Oct 2023stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent analyses of neural networks with shaped activations (i.e. the activation function is scaled as the network size grows) have led to scaling limits described by differential equations. However, these results do not a priori tell us anything about "ordinary" unshaped networks, where the activation is unchanged as the network size grows. In this article, we find similar differential equation based asymptotic characterization for two types of unshaped networks. Firstly, we show that the following two architectures converge to the same infinite-depth-and-width limit at initialization: (i) a fully connected ResNet with a d-1/2 factor on the residual branch, where d is the network depth. (ii) a multilayer perceptron (MLP) with depth d ll width n and shaped ReLU activation at rate d-1/2. Secondly, for an unshaped MLP at initialization, we derive the first order asymptotic correction to the layerwise correlation. In particular, if ρℓ is the correlation at layer ℓ, then qt = ℓ2 (1 - ρℓ) with t = fracℓn converges to an SDE with a singularity at t=0. These results together provide a connection between shaped and unshaped network architectures, and opens up the possibility of studying the effect of normalization methods and how it connects with shaping activation functions.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 30 of its 39 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

30 of 39 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2016Layer Normalizationcited by 5,669
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2017Deep Neural Networks as Gaussian Processescited by 622
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  9. 2021Deep learning: a statistical viewpointcited by 240
  10. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  11. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  12. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
18 more
  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  2. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  3. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  4. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 52
  5. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  6. 2020Stable ResNetcited by 49
  7. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  8. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  9. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  10. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  11. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  12. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  13. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  14. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  15. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  16. 2023Field theory for optimal signal propagation in ResNetscited by 6
  17. 2023Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initializationcited by 3
  18. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2025The Curse of Depth in Large Language Modelscited by 62
  2. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  3. 2026Bayesian Inference with Shaped Deep Non-linear MLPscited by 1
  4. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0