# The critical locus of overparameterized neural networks

arXiv: https://arxiv.org/abs/2005.04210 · HTML: https://mlsanity.com/paper/2005.04210/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2020.

## Built on by (most cited first, 6 of 6)

- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [What Happens after SGD Reaches Zero Loss? --A Mathematical Framework](https://mlsanity.com/paper/2110.06914/) (2021)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [Topological obstruction to the training of shallow ReLU neural networks](https://mlsanity.com/paper/2410.14837/) (2024)
- [How regularization affects the geometry of loss functions](https://mlsanity.com/paper/2307.15744/) (2023)
- [Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations](https://mlsanity.com/paper/2510.24466/) (2025)

## Abstract (arXiv)

Many aspects of the geometry of loss functions in deep learning remain mysterious. In this paper, we work toward a better understanding of the geometry of the loss function L of overparameterized feedforward neural networks. In this setting, we identify several components of the critical locus of L and study their geometric properties. For networks of depth ℓ ≥ 4, we identify a locus of critical points we call the star locus S. Within S we identify a positive-dimensional sublocus C with the property that for p ∈ C, p is a degenerate critical point, and no existing theoretical result guarantees that gradient descent will not converge to p. For very wide networks, we build on earlier work and show that all critical points of L are degenerate, and give lower bounds on the number of zero eigenvalues of the Hessian at each critical point. For networks that are both deep and very wide, we compare the growth rates of the zero eigenspaces of the Hessian at all the different families of critical points that we identify. The results in this paper provide a starting point to a more quantitative understanding of the properties of various components of the critical locus of L.