# Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions

arXiv: https://arxiv.org/abs/2103.10922 · HTML: https://mlsanity.com/paper/2103.10922/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 19 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Theoretical insights into the optimization landscape of over-parameterized shallow neural networks](https://mlsanity.com/paper/1707.04926/) (2017)
- [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017)
- [On the Power of Over-parametrization in Neural Networks with Quadratic Activation](https://mlsanity.com/paper/1803.01206/) (2018)
- [The loss surface of deep and wide neural networks](https://mlsanity.com/paper/1704.08045/) (2017)
- [On the Computational Efficiency of Training Neural Networks](https://mlsanity.com/paper/1410.1141/) (2014)
- [Convergence rates for the stochastic gradient descent method for non-convex objective functions](https://mlsanity.com/paper/1904.01517/) (2019)
- [Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions](https://mlsanity.com/paper/2006.15459/) (2020)
- [On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime](https://mlsanity.com/paper/2005.13530/) (2020)
- [On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics](https://mlsanity.com/paper/2007.15623/) (2020)
- [Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions](https://mlsanity.com/paper/1605.00405/) (2016)
- [No bad local minima: Data independent training error guarantees for multilayer neural networks](https://mlsanity.com/paper/1605.08361/) (2016)
- [On the Quality of the Initial Basin in Overspecified Neural Networks](https://mlsanity.com/paper/1511.04210/) (2015)
- [Exponentially vanishing sub-optimal local minima in multilayer neural networks](https://mlsanity.com/paper/1702.05777/) (2017)
- [A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions](https://mlsanity.com/paper/2102.09924/) (2021)

## Built on by (most cited first, 11 of 11)

- [On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks](https://mlsanity.com/paper/2112.09684/) (2021)
- [Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2107.04479/) (2021)
- [A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions](https://mlsanity.com/paper/2108.04620/) (2021)
- [A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2104.00277/) (2021)
- [Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation](https://mlsanity.com/paper/2108.08106/) (2021)
- [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021)
- [On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems](https://mlsanity.com/paper/2202.12262/) (2022)
- [On the existence of minimizers in shallow residual ReLU neural network optimization landscapes](https://mlsanity.com/paper/2302.14690/) (2023)
- [Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding](https://mlsanity.com/paper/2402.05626/) (2024)
- [Gradient descent provably escapes saddle points in the training of shallow ReLU networks](https://mlsanity.com/paper/2208.02083/) (2022)
- [A Spin Glass Characterization of Neural Networks](https://mlsanity.com/paper/2508.07397/) (2025)

## Abstract (arXiv)

In this paper, we analyze the landscape of the true loss of neural networks with one hidden layer and ReLU, leaky ReLU, or quadratic activation. In all three cases, we provide a complete classification of the critical points in the case where the target function is affine and one-dimensional. In particular, we show that there exist no local maxima and clarify the structure of saddle points. Moreover, we prove that non-global local minima can only be caused by `dead' ReLU neurons. In particular, they do not appear in the case of leaky ReLU or quadratic activation. Our approach is of a combinatorial nature and builds on a careful analysis of the different types of hidden neurons that can occur.