A note on Linear Bottleneck networks and their Transition to Multilinearity
Not on the map. Why?
Abstract (arXiv)
Randomly initialized wide neural networks transition to linear functions of weights as the width grows, in a ball of radius O(1) around initialization. A necessary condition for this result is that all layers of the network are wide enough, i.e., all widths tend to infinity. However, the transition to linearity breaks down when this infinite width assumption is violated. In this work we show that linear networks with a bottleneck layer learn bilinear functions of the weights, in a ball of radius O(1) around initialization. In general, for B-1 bottleneck layers, the network is a degree B multilinear function of weights. Importantly, the degree only depends on the number of bottlenecks and not the total depth of the network.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 30 Jun 2022.
Builds on
19 of 28 references.
References · 19
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
- 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 222
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 184
- 2018Gradient descent aligns the layers of deep linear networkscited by 182
7 more
- 2020On the linearity of large non-linear models: when and why the tangent kernel is constantcited by 129
- 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
- 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
- 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 71
- 2013Phase Retrieval using Alternating Minimizationcited by 58
- 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
- 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
Built on by
Citing papers per year: 2026: 1.
1 paper.