Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learning
Not on the map. Why?
Abstract (arXiv)
We consider gradient-based optimisation of wide, shallow neural networks, where the output of each hidden node is scaled by a positive parameter. The scaling parameters are non-identical, differing from the classical Neural Tangent Kernel (NTK) parameterisation. We prove that for large such neural networks, with high probability, gradient flow and gradient descent converge to a global minimum and can learn features in some sense, unlike in the NTK parameterisation. We perform experiments illustrating our theoretical results and discuss the benefits of such scaling in terms of prunability and transfer learning.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 2 Feb 2023.
Builds on
30 of 38 references.
References · 30
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
18 more
- 2021Deep learning: a statistical viewpointcited by 239
- 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
- 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
- 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
- 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 148
- 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
- 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
- 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
- 2020Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networkscited by 73
- 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 65
- 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
- 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 33
- 2020Stable behaviour of infinitely wide deep neural networkscited by 23
- 2020On the Neural Tangent Kernel of Deep Networks with Orthogonal Initializationcited by 21
- 2021Large-width functional asymptotics for deep Gaussian neural networkscited by 14
- 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
- 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 12
- 2021α-Stable convergence of heavy-tailed infinitely-wide neural networkscited by 5
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.