mlsanityData to 14 Oct 2026

Width and Depth Limits Commute in Residual Networks

arXiv 2302.00453 ↗First posted 1 Feb 2023stat.ML cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

We show that taking the width and depth to infinity in a deep neural network with skip connections, when branches are scaled by 1/√depth (the only nontrivial scaling), result in the same covariance structure no matter how that limit is taken. This explains why the standard infinite-width-then-depth approach provides practical insights even for networks with depth of the same order as width. We also demonstrate that the pre-activations, in this case, have Gaussian distributions which has direct applications in Bayesian deep learning. We conduct extensive simulations that show an excellent match with our theoretical findings.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 1 Feb 2023.

Builds on

33 of 59 references.

References · 33

  1. 2017Deep Neural Networks as Gaussian Processescited by 622
  2. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  3. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  4. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  5. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  6. 2016Deep Information Propagationcited by 195
  7. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  8. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  9. 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
  10. 2020Infinite attention: NNGP and NTK for deep attention networkscited by 121
  11. 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 115
  12. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
21 more
  1. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 102
  2. 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
  3. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  4. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  5. 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
  6. 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 52
  7. 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
  8. 2020Tensor Programs III: Neural Matrix Lawscited by 49
  9. 2020Stable ResNetcited by 49
  10. 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
  11. 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
  12. 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
  13. 2020Robust Pruning at Initializationcited by 31
  14. 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
  15. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  16. 2022On the infinite-depth limit of finite-width neural networkscited by 22
  17. 2019Infinitely deep neural networks as diffusion processescited by 21
  18. 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
  19. 2022Scaling ResNets in the Large-depth Regimecited by 18
  20. 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
  21. 2021Regularization in ResNet with Stochastic Depthcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 3; 2025: 2; 2026: 9.

21 papers.

Sort:

  1. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  2. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  3. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  4. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  5. 2023Simplifying Transformer Blockscited by 37
  6. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  7. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  8. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 20
  9. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  10. 2026On the Residual Scaling of Looped Transformers: Stability and Transferabilitycited by 10
  11. 2026Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuningcited by 9
  12. 2023Differential Equation Scaling Limits of Shaped and Unshaped Neural Networkscited by 4
9 more
  1. 2024Generalization of Scaled Deep ResNets in the Mean-Field Regimecited by 3
  2. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  3. 2023Commutative Width and Depth Scaling in Deep Neural Networkscited by 1
  4. 2026Hyperparameter Transfer Laws for Non-Recurrent Multi-Path Neural Networkscited by 1
  5. 2025Feature Learning Dynamics in Infinite-Depth Neural Networkscited by 0
  6. 2026Collective Kernel EFT for Pre-activation ResNetscited by 0
  7. 2026Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Productcited by 0
  8. 2026Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networkscited by 0
  9. 2026Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradoxcited by 0