mlsanityData to 14 Oct 2026

Improved Convergence Guarantees for Shallow Neural Networks

arXiv 2212.02323 ↗First posted 5 Dec 2022cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We continue a long line of research aimed at proving convergence of depth 2 neural networks, trained via gradient descent, to a global minimum. Like in many previous works, our model has the following features: regression with quadratic loss function, fully connected feedforward architecture, RelU activations, Gaussian data instances and network initialization, adversarial labels. It is more general in the sense that we allow both layers to be trained simultaneously and at em different rates. Our results improve on state-of-the-art [Oymak Soltanolkotabi 20] (training the first layer only) and [Nguyen 21, Section 3.2] (training both layers with Le Cun's initialization). We also report several simple experiments with synthetic data. They strongly suggest that, at least in our model, the convergence phenomenon extends well beyond the ``NTK regime''.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

33 of 44 references.

References · 33

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  4. 2018On Lazy Training in Differentiable Programmingcited by 675
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  8. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  9. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  10. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  11. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  12. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
21 more
  1. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  2. 2020Directional convergence and alignment in deep learningcited by 178
  3. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  4. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  5. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  6. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 121
  7. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  8. 2020Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networkscited by 87
  9. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  10. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  11. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  12. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  13. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 59
  14. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  15. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  16. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  17. 2016Diverse Neural Network Learns True Target Functionscited by 33
  18. 2019Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Networkcited by 27
  19. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  20. 2017Exponentially vanishing sub-optimal local minima in multilayer neural networkscited by 19
  21. 2021Multirate Training of Neural Networkscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networkscited by 5