mlsanityData to 7 Oct 2026

Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activations

arXiv 2102.13233 ↗First posted 25 Feb 2021cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

In this paper, it is shown theoretically that spurious local minima are common for deep fully-connected networks and convolutional neural networks (CNNs) with piecewise linear activation functions and datasets that cannot be fitted by linear models. A motivating example is given to explain the reason for the existence of spurious local minima: each output neuron of deep fully-connected networks and CNNs with piecewise linear activations produces a continuous piecewise linear (CPWL) output, and different pieces of CPWL output can fit disjoint groups of data samples when minimizing the empirical risk. Fitting data samples with different CPWL functions usually results in different levels of empirical risk, leading to prevalence of spurious local minima. This result is proved in general settings with any continuous loss function. The main proof technique is to represent a CPWL function as a maximization over minimization of linear pieces. Deep ReLU networks are then constructed to produce these linear pieces and implement maximization and minimization operations.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 66 of its 92 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

66 of 92 references.

References · 66

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  8. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  12. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
38 more
  1. 2014The Loss Surfaces of Multilayer Networkscited by 278
  2. 2016Deep Learning without Poor Local Minimacited by 233
  3. 2014Qualitatively characterizing neural network optimization problemscited by 186
  4. 2019Complexity of Linear Regions in Deep Networkscited by 166
  5. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  6. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  7. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  8. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  9. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  10. 2016Matrix Completion has No Spurious Local Minimumcited by 124
  11. 2016Identity Matters in Deep Learningcited by 107
  12. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  13. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  14. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  15. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  16. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  17. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 79
  18. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  19. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  20. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  21. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  22. 2019On Connected Sublevel Sets in Deep Learningcited by 51
  23. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  24. 2020On the Number of Linear Regions of Convolutional Neural Networkscited by 49
  25. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  26. 2020Recent advances in deep learning theorycited by 41
  27. 2019SGD Converges to Global Minimum in Deep Learning via Star-convex Pathcited by 39
  28. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
  29. 2017Global optimality conditions for deep neural networkscited by 31
  30. 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 29
  31. 2017Depth Creates No Bad Local Minimacited by 29
  32. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  33. 2018Understanding the Loss Surface of Neural Networks for Binary Classificationcited by 27
  34. 2018Adding One Neuron Can Eliminate All Bad Local Minimacited by 21
  35. 2019Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theorycited by 21
  36. 2015On the Quality of the Initial Basin in Overspecified Neural Networkscited by 20
  37. 2018Spurious Valleys in Two-layer Neural Network Optimization Landscapescited by 20
  38. 2016Local minima in training of neural networkscited by 19

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 2; 2025: 1.

7 papers.

  1. 2022Exact Solutions of a Deep Linear Networkcited by 24
  2. 2024The Real Tropical Geometry of Neural Networkscited by 23
  3. 2024Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networkscited by 9
  4. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
  5. 2025A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorizationcited by 1
  6. 2023Subspace-Configurable Networkscited by 0
  7. 2023Black holes and the loss landscape in machine learningcited by 0