Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks
Not on the map. Why?
Abstract (arXiv)
We study the population loss landscape of two-layer ReLU networks of the form sumk=1K ReLU(wk^top x) in a realisable teacher-student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical organisation of minima into discrete families and shows how overparameterisation changes their stability and reachability under gradient-based dynamics. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 10 Apr 2026.
Builds on
45 of 62 references.
References · 45
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
- 2018A mathematical theory of semantic development in deep neural networkscited by 182
- 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
- 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
- 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
- 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
- 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
- 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
33 more
- 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
- 2018Gradient Descent with Random Initialization: Fast Global Convergence for Nonconvex Phase Retrievalcited by 71
- 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
- 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
- 2023Learning time-scales in two-layers neural networkscited by 57
- 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
- 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
- 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
- 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
- 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
- 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
- 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
- 2019Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Modelscited by 28
- 2021Unveiling the structure of wide flat minima in neural networkscited by 27
- 2019Who is Afraid of Big Bad Minima? Analysis of Gradient-Flow in a Spiked Matrix-Tensor Modelcited by 25
- 2018Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inferencecited by 24
- 2018Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitionscited by 23
- 2019Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activationscited by 22
- 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networkscited by 21
- 2018Spurious Valleys in Two-layer Neural Network Optimization Landscapescited by 20
- 2017The nonsmooth landscape of phase retrievalcited by 16
- 2019Landscape Complexity for the Empirical Risk of Generalized Linear Modelscited by 15
- 2021Gradient Descent on Infinitely Wide Neural Networks: Global Convergence and Generalizationcited by 15
- 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
- 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
- 2019On the Principle of Least Symmetry Breaking in Shallow ReLU Modelscited by 9
- 2023The star-shaped space of solutions of the spherical negative perceptroncited by 8
- 2024Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Trainingcited by 8
- 2024Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergencecited by 8
- 2024The Role of the Time-Dependent Hessian in High-Dimensional Optimizationcited by 7
- 2025A Theory of Initialisation's Impact on Specialisationcited by 6
- 2025Local minima of the empirical risk in high dimension: General theorems and convex examplescited by 4
- 2025Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networkscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.