# The Global Landscape of Neural Networks: An Overview

arXiv: https://arxiv.org/abs/2007.01429 · HTML: https://mlsanity.com/paper/2007.01429/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2020.

## Built on by (most cited first, 30 of 57)

- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifold](https://mlsanity.com/paper/2209.09211/) (2022)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [Taxonomizing local versus global structure in neural network loss landscapes](https://mlsanity.com/paper/2107.11228/) (2021)
- [NTK-SAP: Improving neural network pruning by aligning training dynamics](https://mlsanity.com/paper/2304.02840/) (2023)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Embedding Principle: a hierarchical structure of loss landscape of deep neural networks](https://mlsanity.com/paper/2111.15527/) (2021)
- [A Global Geometric Analysis of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2406.01909/) (2024)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Spatially Varying Nanophotonic Neural Networks](https://mlsanity.com/paper/2308.03407/) (2023)
- [Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm](https://mlsanity.com/paper/2206.02976/) (2022)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [When Expressivity Meets Trainability: Fewer than n Neurons Can Work](https://mlsanity.com/paper/2210.12001/) (2022)
- [Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions](https://mlsanity.com/paper/2301.07966/) (2023)
- [Type-II Saddles and Probabilistic Stability of Stochastic Gradient Descent](https://mlsanity.com/paper/2303.13093/) (2023)
- [Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations](https://mlsanity.com/paper/2102.13233/) (2021)
- [Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data](https://mlsanity.com/paper/2305.07845/) (2023)
- [A Tree-Structured Multi-Task Model Recommender](https://mlsanity.com/paper/2203.05092/) (2022)
- [Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networks](https://mlsanity.com/paper/2206.02139/) (2022)
- [Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets](https://mlsanity.com/paper/2111.14039/) (2021)
- [Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization](https://mlsanity.com/paper/2309.00508/) (2023)
- [Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks](https://mlsanity.com/paper/2204.07261/) (2022)
- [Deep Loss Convexification for Learning Iterative Models](https://mlsanity.com/paper/2411.10649/) (2024)
- [Flat Channels to Infinity in Neural Loss Landscapes](https://mlsanity.com/paper/2506.14951/) (2025)
- [Exploring Landscapes for Better Minima along Valleys](https://mlsanity.com/paper/2510.27153/) (2025)
- [Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks](https://mlsanity.com/paper/2111.12963/) (2021)
- [Blind Estimation of a Doubly Selective OFDM Channel: A Deep Learning Algorithm and Theory](https://mlsanity.com/paper/2206.07483/) (2022)
- [Invariant Aggregator for Defending against Federated Backdoor Attacks](https://mlsanity.com/paper/2210.01834/) (2022)
- [Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks](https://mlsanity.com/paper/2405.17501/) (2024)
- [AdapMTL: Adaptive Pruning Framework for Multitask Learning Model](https://mlsanity.com/paper/2408.03913/) (2024)

## Abstract (arXiv)

One of the major concerns for neural network training is that the non-convexity of the associated loss functions may cause bad landscape. The recent success of neural networks suggests that their loss landscape is not too bad, but what specific results do we know about the landscape? In this article, we review recent findings and results on the global landscape of neural networks. First, we point out that wide neural nets may have sub-optimal local minima under certain assumptions. Second, we discuss a few rigorous results on the geometric properties of wide networks such as "no bad basin", and some modifications that eliminate sub-optimal local minima and/or decreasing paths to infinity. Third, we discuss visualization and empirical explorations of the landscape for practical neural nets. Finally, we briefly discuss some convergence results and their relation to landscape results.