Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
Not on the map. Why?
Abstract (arXiv)
We analyze the learning dynamics of infinitely wide neural networks with a finite sized bottle-neck. Unlike the neural tangent kernel limit, a bottleneck in an otherwise infinite width network al-lows data dependent feature learning in its bottle-neck representation. We empirically show that a single bottleneck in infinite networks dramatically accelerates training when compared to purely in-finite networks, with an improved overall performance. We discuss the acceleration phenomena by drawing similarities to infinitely wide deep linear models, where the acceleration effect of a bottleneck can be understood theoretically.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 22 references that are in the map. First posted on arXiv 1 Jul 2021.
Builds on
16 of 22 references.
References · 16
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
- 2020Feature Learning in Infinite-Width Neural Networkscited by 182
- 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 164
- 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
- 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 139
- 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
- 2021Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamicscited by 57
- 2020Tensor Programs III: Neural Matrix Lawscited by 47
- 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
4 more
- 2019Addressing the Loss-Metric Mismatch with Adaptive Loss Alignmentcited by 33
- 2015Steps Toward Deep Kernel Methods from Infinite Neural Networkscited by 28
- 2020Wide Neural Networks with Bottlenecks are Deep Gaussian Processescited by 18
- 2020On Infinite-Width Hypernetworkscited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.