mlsanityData to 7 Oct 2026

On the infinite-depth limit of finite-width neural networks

arXiv 2210.00688 ↗First posted 3 Oct 2022stat.ML cs.LG math.PRcited by 22

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the infinite-depth limit of finite-width residual neural networks with random Gaussian weights. With proper scaling, we show that by fixing the width and taking the depth to infinity, the pre-activations converge in distribution to a zero-drift diffusion process. Unlike the infinite-width limit where the pre-activation converge weakly to a Gaussian random variable, we show that the infinite-depth limit yields different distributions depending on the choice of the activation function. We document two cases where these distributions have closed-form (different) expressions. We further show an intriguing change of regime phenomenon of the post-activation norms when the width increases from 3 to 4. Lastly, we study the sequential limit infinite-depth-then-infinite-width and compare it with the more commonly studied infinite-width-then-infinite-depth limit.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 3 Oct 2022.

Builds on

23 of 39 references.

References · 23

  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  3. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  4. 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
  5. 2016Deep Information Propagationcited by 194
  6. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
  7. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
  8. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  9. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  10. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  11. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  12. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
11 more
  1. 2020Stable ResNetcited by 49
  2. 2020Tensor Programs III: Neural Matrix Lawscited by 47
  3. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  4. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  5. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  6. 2020Robust Pruning at Initializationcited by 31
  7. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  8. 2019Infinitely deep neural networks as diffusion processescited by 21
  9. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  10. 2022Scaling ResNets in the Large-depth Regimecited by 18
  11. 2021Regularization in ResNet with Stochastic Depthcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 6; 2025: 1; 2026: 5.

22 papers.

Sort:

  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  2. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 64
  3. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  4. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  5. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 42
  6. 2024Infinite Limits of Multi-head Transformer Dynamicscited by 38
  7. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  8. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  9. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  10. 2023Width and Depth Limits Commute in Residual Networkscited by 21
  11. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  12. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
10 more
  1. 2024Theoretical Insights into Fine-Tuning Attention Mechanism: Generalization and Optimizationcited by 9
  2. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  3. 2023Differential Equation Scaling Limits of Shaped and Unshaped Neural Networkscited by 4
  4. 2025Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Sizecited by 4
  5. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  6. 2026Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networkscited by 0
  7. 2026Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptationcited by 0
  8. 2026Chemical Reaction Networks Learn Better than Spiking Neural Networkscited by 0
  9. 2026How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPscited by 0
  10. 2026Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networkscited by 0