mlsanityData to 7 Oct 2026

The Global Landscape of Neural Networks: An Overview

arXiv 2007.01429 ↗First posted 2 Jul 2020cs.LG math.OC stat.MLcited by 57

Not on the map. Why?

Abstract (arXiv)

One of the major concerns for neural network training is that the non-convexity of the associated loss functions may cause bad landscape. The recent success of neural networks suggests that their loss landscape is not too bad, but what specific results do we know about the landscape? In this article, we review recent findings and results on the global landscape of neural networks. First, we point out that wide neural nets may have sub-optimal local minima under certain assumptions. Second, we discuss a few rigorous results on the geometric properties of wide networks such as "no bad basin", and some modifications that eliminate sub-optimal local minima and/or decreasing paths to infinity. Third, we discuss visualization and empirical explorations of the landscape for practical neural nets. Finally, we briefly discuss some convergence results and their relation to landscape results.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 13; 2023: 11; 2024: 9; 2025: 9; 2026: 5.

57 papers.

Sort:

  1. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  2. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  3. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  4. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  5. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  6. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  7. 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 21
  8. 2024A Global Geometric Analysis of Maximal Coding Rate Reductioncited by 13
  9. 2023Spatially Varying Nanophotonic Neural Networkscited by 12
  10. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  11. 2022Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithmcited by 9
  12. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
38 more
  1. 2022When Expressivity Meets Trainability: Fewer than n Neurons Can Workcited by 8
  2. 2023Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regionscited by 8
  3. 2023Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descentcited by 8
  4. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  5. 2023Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Datacited by 7
  6. 2022A Tree-Structured Multi-Task Model Recommendercited by 6
  7. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  8. 2021Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Netscited by 4
  9. 2023Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterizationcited by 4
  10. 2022Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networkscited by 3
  11. 2024Deep Loss Convexification for Learning Iterative Modelscited by 3
  12. 2025Flat Channels to Infinity in Neural Loss Landscapescited by 3
  13. 2025Exploring Landscapes for Better Minima along Valleyscited by 3
  14. 2021Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networkscited by 2
  15. 2022Blind Estimation of a Doubly Selective OFDM Channel: A Deep Learning Algorithm and Theorycited by 2
  16. 2022Invariant Aggregator for Defending against Federated Backdoor Attackscited by 2
  17. 2024Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networkscited by 2
  18. 2024AdapMTL: Adaptive Pruning Framework for Multitask Learning Modelcited by 2
  19. 2026On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvaturecited by 2
  20. 2021WGAN with an Infinitely Wide Generator Has No Spurious Stationary Pointscited by 1
  21. 2021Robustness-via-Synthesis: Robust Training with Generative Adversarial Perturbationscited by 1
  22. 2022Parameter Convex Neural Networkscited by 1
  23. 2023Training a Two Layer ReLU Network Analyticallycited by 1
  24. 2023An Adaptive Tangent Feature Perspective of Neural Networkscited by 1
  25. 2024Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networkscited by 1
  26. 2024Component-based Sketching for Deep ReLU Netscited by 1
  27. 2025Collapsing Taylor Mode Automatic Differentiationcited by 1
  28. 2025A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorizationcited by 1
  29. 2026Noise-Driven Exploration and Transient Freezing Select Flat Minima in Stochastic Gradient Descentcited by 1
  30. 2026Visualizing the loss landscapes of physics-informed neural networkscited by 1
  31. 2026Unveiling Hidden Convexity in Deep Learning: a Sparse Signal Processing Perspectivecited by 1
  32. 2021Neural Networks with Complex-Valued Weights Have No Spurious Local Minimacited by 0
  33. 2022Global Convergence Analysis of Deep Linear Networks with A One-neuron Layercited by 0
  34. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0
  35. 2023Physics Inspired Approaches To Understanding Gaussian Processescited by 0
  36. 2023Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Losscited by 0
  37. 2023Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Methodcited by 0
  38. 2024Manifold Metric: A Loss Landscape Approach for Predicting Model Performancecited by 0

50 most cited of 57 All 57 (JSON)