mlsanityData to 13 Oct 2026

Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions

arXiv 2103.10922 ↗First posted 19 Mar 2021cs.LG cs.NA math.NAcited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we analyze the landscape of the true loss of neural networks with one hidden layer and ReLU, leaky ReLU, or quadratic activation. In all three cases, we provide a complete classification of the critical points in the case where the target function is affine and one-dimensional. In particular, we show that there exist no local maxima and clarify the structure of saddle points. Moreover, we prove that non-global local minima can only be caused by `dead' ReLU neurons. In particular, they do not appear in the case of leaky ReLU or quadratic activation. Our approach is of a combinatorial nature and builds on a careful analysis of the different types of hidden neurons that can occur.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 19 Mar 2021.

Builds on

20 of 36 references.

References · 20

  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 786
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 288
  4. 2014The Loss Surfaces of Multilayer Networkscited by 280
  5. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  6. 2016Deep Learning without Poor Local Minimacited by 234
  7. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  8. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  9. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  10. 2017The loss surface of deep and wide neural networkscited by 74
  11. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  12. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
8 more
  1. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  2. 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
  3. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 38
  4. 2016Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regionscited by 34
  5. 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 29
  6. 2015On the Quality of the Initial Basin in Overspecified Neural Networkscited by 20
  7. 2017Exponentially vanishing sub-optimal local minima in multilayer neural networkscited by 19
  8. 2021A proof of convergence for gradient descent in the training of artificial neural networks for constant target functionscited by 14

Built on by

Cited by year

Citing papers per year: 2021: 6; 2022: 2; 2023: 1; 2024: 1; 2025: 1.

11 papers.

  1. 2021On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networkscited by 12
  2. 2021Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activationcited by 11
  3. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  4. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  5. 2021Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activationcited by 9
  6. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  7. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
  8. 2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
  9. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  10. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  11. 2025A Spin Glass Characterization of Neural Networkscited by 1