# The loss landscape of overparameterized neural networks

arXiv: https://arxiv.org/abs/1804.10200 · HTML: https://mlsanity.com/paper/1804.10200/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2018.

## Built on by (most cited first, 30 of 39)

- [What Happens after SGD Reaches Zero Loss? --A Mathematical Framework](https://mlsanity.com/paper/2110.06914/) (2021)
- [Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis](https://mlsanity.com/paper/2105.01650/) (2021)
- [Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalization](https://mlsanity.com/paper/2307.11007/) (2023)
- [Symmetry in Neural Network Parameter Spaces](https://mlsanity.com/paper/2506.13018/) (2025)
- [Symmetries, flat minima, and the conserved quantities of gradient flow](https://mlsanity.com/paper/2210.17216/) (2022)
- [Taxonomizing local versus global structure in neural network loss landscapes](https://mlsanity.com/paper/2107.11228/) (2021)
- [Improving Generalization and Convergence by Enhancing Implicit Regularization](https://mlsanity.com/paper/2405.20763/) (2024)
- [On minimal representations of shallow ReLU networks](https://mlsanity.com/paper/2108.05643/) (2021)
- [From Activation to Initialization: Scaling Insights for Optimizing Neural Fields](https://mlsanity.com/paper/2403.19205/) (2024)
- [Deep Learning Generalization and the Convex Hull of Training Sets](https://mlsanity.com/paper/2101.09849/) (2021)
- [Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity](https://mlsanity.com/paper/2404.06391/) (2024)
- [Double descent in quantum kernel methods](https://mlsanity.com/paper/2501.10077/) (2025)
- [Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning](https://mlsanity.com/paper/2501.15556/) (2025)
- [A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models](https://mlsanity.com/paper/2506.02269/) (2025)
- [Tilting the playing field: Dynamical loss functions for machine learning](https://mlsanity.com/paper/2102.03793/) (2021)
- [Singular-limit analysis of gradient descent with noise injection](https://mlsanity.com/paper/2404.12293/) (2024)
- [Topological obstruction to the training of shallow ReLU neural networks](https://mlsanity.com/paper/2410.14837/) (2024)
- [Variational Quantum Classifiers Through the Lens of the Hessian](https://mlsanity.com/paper/2105.10162/) (2021)
- [Careful with that Scalpel: Improving Gradient Surgery with an EMA](https://mlsanity.com/paper/2402.02998/) (2024)
- [Minimum norm interpolation by perceptra: Explicit regularization and implicit bias](https://mlsanity.com/paper/2311.06138/) (2023)
- [On the Convergence of Gradient Descent for Large Learning Rates](https://mlsanity.com/paper/2402.13108/) (2024)
- [Deep Loss Convexification for Learning Iterative Models](https://mlsanity.com/paper/2411.10649/) (2024)
- [Flat Channels to Infinity in Neural Loss Landscapes](https://mlsanity.com/paper/2506.14951/) (2025)
- [FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks](https://mlsanity.com/paper/2204.04511/) (2022)
- [Hierarchical Weight Averaging for Deep Neural Networks](https://mlsanity.com/paper/2304.11519/) (2023)
- [Global L2 minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning](https://mlsanity.com/paper/2311.15487/) (2023)
- [Enhancing material property prediction with ensemble deep graph convolutional networks](https://mlsanity.com/paper/2407.18847/) (2024)
- [Gradient flow in parameter space is equivalent to linear interpolation in output space](https://mlsanity.com/paper/2408.01517/) (2024)
- [Visualizing the loss landscapes of physics-informed neural networks](https://mlsanity.com/paper/2602.05849/) (2026)
- [SING: A Plug-and-Play DNN Learning Technique](https://mlsanity.com/paper/2305.15997/) (2023)

## Abstract (arXiv)

We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori one might imagine that the loss function looks like a typical function from ℝn to ℝ - in particular, nonconvex, with discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has n parameters and is trained on d data points, with n>d, we show that the locus M of global minima of L is usually not discrete, but rather an n-d dimensional submanifold of ℝn. In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that M is typically a very high-dimensional subset of ℝn.