mlsanityData to 14 Oct 2026

Theoretical insights into the optimization landscape of over-parameterized shallow neural networks

arXiv 1707.04926 ↗First posted 16 Jul 2017cs.LG cs.IT math.ITcited by 128

Not on the map. Why?

Abstract (arXiv)

In this paper we study the problem of learning a shallow artificial neural network that best fits a training data set. We study this problem in the over-parameterized regime where the number of observations are fewer than the number of parameters in the model. We show that with quadratic activations the optimization landscape of training such shallow neural networks has certain favorable characteristics that allow globally optimal models to be found efficiently using a variety of local search heuristics. This result holds for an arbitrary training data of input/output pairs. For differentiable activation functions we also show that gradient descent, when suitably initialized, converges at a linear rate to a globally optimal model. This result focuses on a realizable model where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted weight coefficients.

Key facts. Cited by 128 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 34; 2022: 35; 2023: 24; 2024: 11; 2025: 18; 2026: 6.

128 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  3. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  4. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  5. 2021Tensor Methods in Computer Vision and Deep Learningcited by 52
  6. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  7. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  8. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  9. 2021Constrained Learning with Non-Convex Lossescited by 42
  10. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  11. 2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
  12. 2022Probabilistically Robust Learning: Balancing Average- and Worst-case Performancecited by 37
38 more
  1. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  2. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  3. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  4. 2023The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensingcited by 33
  5. 2022Green Learning: Introduction, Examples and Outlookcited by 30
  6. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  7. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  8. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 28
  9. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  10. 2022Convergence of gradient descent for deep neural networkscited by 24
  11. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  12. 2022Theoretical Perspectives on Deep Learning Methods in Inverse Problemscited by 22
  13. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  14. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  15. 2023Generalization and Stability of Interpolating Neural Networks with Minimal Widthcited by 21
  16. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  17. 2021Learning Transferable Parameters for Unsupervised Domain Adaptationcited by 19
  18. 2022How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysiscited by 19
  19. 2023Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networkscited by 19
  20. 2023On the Convergence and Sample Complexity Analysis of Deep Q-Networks with ε-Greedy Explorationcited by 17
  21. 2025Optimizers Qualitatively Alter Solutions And We Should Leverage Thiscited by 17
  22. 2023Traffic Prediction using Artificial Intelligence: Review of Recent Advances and Emerging Opportunitiescited by 16
  23. 2022Tensor-on-Tensor Regression: Riemannian Optimization, Over-parameterization, Statistical-computational Gap, and Their Interplaycited by 15
  24. 2023Resilient Constrained Learningcited by 15
  25. 2021Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Timecited by 14
  26. 2021Understanding Overparameterization in Generative Adversarial Networkscited by 14
  27. 2021Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networkscited by 14
  28. 2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 13
  29. 2022Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomialscited by 13
  30. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  31. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 13
  32. 2021Fundamental Tradeoffs in Distributionally Adversarial Trainingcited by 12
  33. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 12
  34. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  35. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  36. 2022On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimescited by 11
  37. 2022Non-convex online learning via algorithmic equivalencecited by 11
  38. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11

50 most cited of 128 All 128 (JSON)