Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization
Not on the map. Why?
Abstract (arXiv)
Despite remarkable performance on a variety of tasks, many properties of deep neural networks are not yet theoretically understood. One such mystery is the depth degeneracy phenomenon: the deeper you make your network, the closer your network is to a constant function on initialization. In this paper, we examine the evolution of the angle between two inputs to a ReLU neural network as a function of the number of layers. By using combinatorial expansions, we find precise formulas for how fast this angle goes to zero as depth increases. These formulas capture microscopic fluctuations that are not visible in the popular framework of infinite width limits, and leads to qualitatively different predictions. We validate our theoretical results with Monte Carlo experiments and show that our results accurately approximate finite network behaviour. reviewWe also empirically investigate how the depth degeneracy phenomenon can negatively impact training of real networks. The formulas are given in terms of the mixed moments of correlated Gaussians passed through the ReLU function. We also find a surprising combinatorial connection between these mixed moments and the Bessel numbers that allows us to explicitly evaluate these moments.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 12 of its 25 references that are in the map. First posted on arXiv 20 Feb 2023.
Builds on
12 of 25 references.
References · 12
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
- 2016Deep Information Propagationcited by 194
- 2015The Power of Depth for Feedforward Neural Networkscited by 181
- 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
- 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
- 2021Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shapingcited by 51
- 2022The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initializationcited by 41
- 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
- 2020Deep Networks and the Multiple Manifold Problemcited by 35
- 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
Built on by
Citing papers per year: 2023: 3.
3 papers.