Commutative Width and Depth Scaling in Deep Neural Networks
Not on the map. Why?
Abstract (arXiv)
This paper is the second in the series Commutative Scaling of Width and Depth (WD) about commutativity of infinite width and depth limits in deep neural networks. Our aim is to understand the behaviour of neural functions (functions that depend on a neural network model) as width and depth go to infinity (in some sense), and eventually identify settings under which commutativity holds, i.e. the neural function tends to the same limit no matter how width and depth limits are taken. In this paper, we formally introduce and define the commutativity framework, and discuss its implications on neural network design and scaling. We study commutativity for the neural covariance kernel which reflects how network layers separate data. Our findings extend previous results established in [55] by showing that taking the width and depth to infinity in a deep neural network with skip connections, when branches are suitably scaled to avoid exploding behaviour, result in the same covariance structure no matter how that limit is taken. This has a number of theoretical and practical implications that we discuss in the paper. The proof techniques in this paper are novel and rely on tools that are more accessible to readers who are not familiar with stochastic calculus (used in the proofs of WD(I))).
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 39 of its 64 references that are in the map. First posted on arXiv 2 Oct 2023.
Builds on
39 of 64 references.
References · 39
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
- 2016Deep Information Propagationcited by 194
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
- 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
- 2020Infinite attention: NNGP and NTK for deep attention networkscited by 117
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
- 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
- 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
27 more
- 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
- 2020Bayesian Deep Ensembles via the Neural Tangent Kernelcited by 90
- 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
- 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
- 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
- 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
- 2018Products of Many Large Random Matrices and Gradients in Deep Neural Networkscited by 49
- 2020Stable ResNetcited by 49
- 2020Tensor Programs III: Neural Matrix Lawscited by 47
- 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
- 2020Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networkscited by 44
- 2019Mean Field Analysis of Deep Neural Networkscited by 42
- 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
- 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
- 2021The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initializationcited by 33
- 2020Robust Pruning at Initializationcited by 31
- 2021Precise characterization of the prior predictive distribution of deep ReLU networkscited by 28
- 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
- 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
- 2022On the infinite-depth limit of finite-width neural networkscited by 22
- 2019Infinitely deep neural networks as diffusion processescited by 21
- 2023Width and Depth Limits Commute in Residual Networkscited by 21
- 2022The Influence of Learning Rule on Representation Dynamics in Wide Neural Networkscited by 19
- 2021Feature Learning and Signal Propagation in Deep Neural Networkscited by 18
- 2022Gaussian Pre-Activations in Neural Networks: Myth or Reality?cited by 9
- 2021Regularization in ResNet with Stochastic Depthcited by 7
- 2023Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initializationcited by 3
Built on by
Citing papers per year: 2023: 1.
1 paper.