mlsanityData to 7 Oct 2026

On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems

arXiv 2202.12262 ↗First posted 23 Feb 2022cs.LG math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

We study the loss landscape of training problems for deep artificial neural networks with a one-dimensional real output whose activation functions contain an affine segment and whose hidden layers have width at least two. It is shown that such problems possess a continuum of spurious (i.e., not globally optimal) local minima for all target functions that are not affine. In contrast to previous works, our analysis covers all sampling and parameterization regimes, general differentiable loss functions, arbitrary continuous nonpolynomial activation functions, and both the finite- and infinite-dimensional setting. It is further shown that the appearance of the spurious local minima in the considered training problems is a direct consequence of the universal approximation theorem and that the underlying mechanisms also cause, e.g., Lp-best approximation problems to be ill-posed in the sense of Hadamard for all networks that do not have a dense image. The latter result also holds without the assumption of local affine linearity and without any conditions on the hidden layers.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 23 Feb 2022.

Builds on

33 of 59 references.

References · 33

  1. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  4. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  5. 2016Deep Learning without Poor Local Minimacited by 233
  6. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  7. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  8. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  9. 2019Optimization for deep learning: theory and algorithmscited by 70
  10. 2021The Modern Mathematics of Deep Learningcited by 63
  11. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  12. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
21 more
  1. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  2. 2020On the Global Convergence of Training Deep Linear ResNetscited by 29
  3. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  4. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 22
  5. 2019Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theorycited by 21
  6. 2016Local minima in training of neural networkscited by 19
  7. 2020Training Linear Neural Networks: Non-Local Convergence and Complexity Resultscited by 19
  8. 2020Piecewise linear activations substantially shape the loss surfaces of neural networkscited by 19
  9. 2018On the Benefit of Width for Neural Networks: Disappearance of Bad Basinscited by 16
  10. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  11. 2020Non-convergence of stochastic gradient descent in the training of deep neural networkscited by 14
  12. 2018PLU: The Piecewise Linear Unit Activation Functioncited by 11
  13. 2020Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidancecited by 11
  14. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  15. 2018Non-attracting Regions of Local Minima in Deep and Wide Neural Networkscited by 8
  16. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  17. 2020Training Two-Layer ReLU Networks with Gradient Descent is Inconsistentcited by 6
  18. 2020The critical locus of overparameterized neural networkscited by 6
  19. 2019Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activationscited by 4
  20. 2020On the Nonuniqueness and Instability of Solutions of Tracking-Type Optimal Control Problemscited by 2
  21. 2020On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 1; 2026: 1.

8 papers.

  1. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  2. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  3. 2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
  4. 2025Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization in the training of deep neural networkscited by 6
  5. 2023On the existence of optimal shallow feedforward networks with ReLU activationcited by 5
  6. 2023On the Identification and Optimization of Nonsmooth Superposition Operators in Semilinear Elliptic PDEscited by 2
  7. 2024Learnability of high-dimensional targets by two-parameter models and gradient flowcited by 0
  8. 2026Benign Loss Landscapes Can Coexist with Worst-Case Hardnesscited by 0