A Geometric Characterization of the Stationary Plateau for Two-Layer Neural Networks
Not on the map. Why?
Abstract (arXiv)
We investigate the geometric structure of stationary plateaus that arise in the loss landscape of two-layer neural networks with smooth activation functions. We focus on the phenomenon of "neuron splitting" where duplicating a hidden neuron yields an affine set of stationary points in a wider network. We provide a comprehensive classification of all stationary points on these plateaus, determining under what conditions they constitute local minima or saddle points. Our characterization hinges on a per-neuron curvature object we term the "inner Hessian" matrix. Our analysis reveals that the definiteness of the inner Hessian and the choice of splitting coefficients jointly dictate the local geometry of the plateau. We show that "splitting" a local minimum can yield either a mixture of local minima and saddles or an all-saddle plateau, with a concrete sure-saddle region identified under mild assumptions. In contrast, splitting a saddle point always produces a plateau of saddle points. Our results unify and extend prior landscape analyses, elucidating when and how model expansion preserves or alters the nature of stationary points. These findings offer new geometric insights into the effects of width expansion and reparameterization in neural networks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 3 Jun 2026.
Builds on
26 of 33 references.
References · 26
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
- 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
- 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
- 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
- 2016Deep Learning without Poor Local Minimacited by 234
- 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 127
- 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
- 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
- 2024Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspectivecited by 81
14 more
- 2017The loss surface of deep and wide neural networkscited by 74
- 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 65
- 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 63
- 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
- 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
- 2019Semi-flat minima and saddle points by embedding neural networks to overparameterizationcited by 30
- 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
- 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
- 2022The effect of smooth parametrizations on nonconvex optimization landscapescited by 21
- 2020Piecewise linear activations substantially shape the loss surfaces of neural networkscited by 19
- 2018On the Benefit of Width for Neural Networks: Disappearance of Bad Basinscited by 16
- 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
- 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
- 2024Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networkscited by 2
Built on by
Not cited yet.