mlsanityData to 7 Oct 2026

The loss surface of deep and wide neural networks

arXiv 1704.08045 ↗First posted 26 Apr 2017cs.LG cs.AI cs.CVcited by 74

Not on the map. Why?

Abstract (arXiv)

While the optimization problem behind deep neural networks is highly non-convex, it is frequently observed in practice that training deep networks seems possible without getting stuck in suboptimal points. It has been argued that this is the case as all local minima are close to being globally optimal. We show that this is (almost) true, in fact almost all local minima are globally optimal, for a fully connected network with squared loss and analytic activation function given that the number of hidden units of one layer of the network is larger than the number of training points and the network structure from this layer on is pyramidal.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 12; 2023: 9; 2024: 10; 2025: 10; 2026: 6.

74 papers.

Sort:

  1. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  2. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  3. 2022Mechanistic Mode Connectivitycited by 50
  4. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  5. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  6. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  7. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  8. 2023Revisiting the Noise Model of Stochastic Gradient Descentcited by 35
  9. 2021Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labelscited by 32
  10. 2023SI-MIL: Taming Deep MIL for Self-Interpretability in Gigapixel Histopathologycited by 25
  11. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  12. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
38 more
  1. 2021Exponentially Many Local Minima in Quantum Neural Networkscited by 22
  2. 2022Neural Tangent Kernel: A Surveycited by 22
  3. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  4. 2021Noether: The More Things Change, the More Stay the Samecited by 19
  5. 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
  6. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  7. 2021Trusted Artificial Intelligence: Towards Certification of Machine Learning Applicationscited by 13
  8. 2021Deep Network Approximation in Terms of Intrinsic Parameterscited by 12
  9. 2023Interpretability of Machine Learning: Recent Advances and Future Prospectscited by 12
  10. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  11. 2023How Spurious Features Are Memorized: Precise Analysis for Random and NTK Featurescited by 11
  12. 2021When Are Solutions Connected in Deep Networks?cited by 10
  13. 2021On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regimecited by 10
  14. 2022Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Networkcited by 10
  15. 2023Data efficiency and extrapolation trends in neural network interatomic potentialscited by 10
  16. 2022A Kernel-Expanded Stochastic Neural Networkcited by 9
  17. 2021A Note on Connectivity of Sublevel Sets in Deep Learningcited by 8
  18. 2021A singular Riemannian geometry approach to Deep Neural Networks II. Reconstruction of 1-D equivalence classescited by 8
  19. 2021Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Spacecited by 7
  20. 2021Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibrationcited by 7
  21. 2022Piecewise Linear Neural Networks and Deep Learningcited by 7
  22. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  23. 2024Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activationscited by 4
  24. 2024Calabi-Yau metrics through Grassmannian learning and Donaldson's algorithmcited by 4
  25. 2025Explaining Grokking and Information Bottleneck through Neural Collapse Emergencecited by 4
  26. 2021Achieving Small Test Error in Mildly Overparameterized Neural Networkscited by 3
  27. 2021Theoretical Exploration of Flexible Transmitter Modelcited by 3
  28. 2022Wavelet Regularization Benefits Adversarial Trainingcited by 3
  29. 2023Gradient is All You Need? How Consensus-Based Optimization can be Interpreted as a Stochastic Relaxation of Gradient Descentcited by 3
  30. 2023Sparse Deep Learning for Time Series Data: Theory and Applicationscited by 3
  31. 2025Connecting Independently Trained Modes via Layer-Wise Connectivitycited by 3
  32. 2025Flat Channels to Infinity in Neural Loss Landscapescited by 3
  33. 2021Analysis and Optimisation of Bellman Residual Errors with Neural Function Approximationcited by 2
  34. 2024Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networkscited by 2
  35. 2021Overestimation learning with guaranteescited by 1
  36. 2022Parameter Convex Neural Networkscited by 1
  37. 2022MAC: A Meta-Learning Approach for Feature Learning and Recombinationcited by 1
  38. 2024A topological description of loss surfaces based on Betti Numberscited by 1

50 most cited of 74 All 74 (JSON)