mlsanityData to 14 Oct 2026

Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding

arXiv 2402.05626 ↗First posted 8 Feb 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we study the loss landscape of one-hidden-layer neural networks with ReLU-like activation functions trained with the empirical squared loss using gradient descent (GD). We identify the stationary points of such networks, which significantly slow down loss decrease during training. To capture such points while accounting for the non-differentiability of the loss, the stationary points that we study are directional stationary points, rather than other notions like Clarke stationary points. We show that, if a stationary point does not contain "escape neurons", which are defined with first-order conditions, it must be a local minimum. Moreover, for the scalar-output case, the presence of an escape neuron guarantees that the stationary point is not a local minimum. Our results refine the description of the saddle-to-saddle training process starting from infinitesimally small (vanishing) initialization for shallow ReLU-like networks: By precluding the saddle escape types that previous works did not rule out, we advance one step closer to a complete picture of the entire dynamics. Moreover, we are also able to fully discuss how network embedding, which is to instantiate a narrower network with a wider network, reshapes the stationary points.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

39 of 51 references.

References · 39

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2018On Lazy Training in Differentiable Programmingcited by 675
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  4. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  5. 2017How to Escape Saddle Points Efficientlycited by 285
  6. 2016Deep Learning without Poor Local Minimacited by 234
  7. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  8. 2017Practical Gauss-Newton Optimisation for Deep Learningcited by 142
  9. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  10. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
  11. 2018Stochastic subgradient method converges on tame functionscited by 124
  12. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
27 more
  1. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 76
  2. 2017The loss surface of deep and wide neural networkscited by 74
  3. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  4. 2020Phase diagram for two-layer ReLU neural networks at infinite-width limitcited by 67
  5. 2018Escaping Saddles with Stochastic Gradientscited by 66
  6. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  7. 2019Splitting Steepest Descent for Growing Neural Architecturescited by 55
  8. 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
  9. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  10. 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 30
  11. 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 30
  12. 2020The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutionscited by 30
  13. 2016Gradient Descent Converges to Minimizerscited by 25
  14. 2023LEMON: Lossless model expansioncited by 23
  15. 2015On the Quality of the Initial Basin in Overspecified Neural Networkscited by 20
  16. 2017Exponentially vanishing sub-optimal local minima in multilayer neural networkscited by 19
  17. 2020Piecewise linear activations substantially shape the loss surfaces of neural networkscited by 19
  18. 2024Early alignment in two-layer networks training is a two-edged swordcited by 18
  19. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  20. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  21. 2017First-order Methods Almost Always Avoid Saddle Pointscited by 14
  22. 2020Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamicscited by 13
  23. 2021Landscape analysis for shallow neural networks: complete classification of critical points for affine target functionscited by 11
  24. 2021SGD with a Constant Large Learning Rate Can Converge to Local Maximacited by 8
  25. 2023Should Under-parameterized Student Networks Copy or Average Teacher Weights?cited by 8
  26. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  27. 2018Efficiently testing local optimality and escaping saddles for ReLU networkscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  2. 2026From Approximation Rates to Loss-Landscape Barrier Decay in Shallow ReLU Networkscited by 0
  3. 2026A Theory of Saddle Escape in Deep Nonlinear Networkscited by 0
  4. 2026A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networkscited by 0
  5. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0
  6. 2026Removing spurious minima for planar features by skip connectionscited by 0