mlsanityData to 7 Oct 2026

The loss landscape of overparameterized neural networks

arXiv 1804.10200 ↗First posted 26 Apr 2018cs.LG cs.AI cs.NEcited by 38

Not on the map. Why?

Abstract (arXiv)

We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori one might imagine that the loss function looks like a typical function from ℝn to ℝ - in particular, nonconvex, with discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has n parameters and is trained on d data points, with n>d, we show that the locus M of global minima of L is usually not discrete, but rather an n-d dimensional submanifold of ℝn. In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that M is typically a very high-dimensional subset of ℝn.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 2; 2023: 6; 2024: 13; 2025: 8; 2026: 2.

38 papers.

Sort:

  1. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  2. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  3. 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 38
  4. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  5. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  6. 2025Symmetry in Neural Network Parameter Spacescited by 31
  7. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  8. 2021On minimal representations of shallow ReLU networkscited by 13
  9. 2024From Activation to Initialization: Scaling Insights for Optimizing Neural Fieldscited by 11
  10. 2021Deep Learning Generalization and the Convex Hull of Training Setscited by 10
  11. 2024Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivitycited by 9
  12. 2025Double descent in quantum kernel methodscited by 9
26 more
  1. 2025Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learningcited by 9
  2. 2025A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Modelscited by 8
  3. 2021Tilting the playing field: Dynamical loss functions for machine learningcited by 7
  4. 2024Topological obstruction to the training of shallow ReLU neural networkscited by 6
  5. 2024Singular-limit analysis of gradient descent with noise injectioncited by 5
  6. 2021Variational Quantum Classifiers Through the Lens of the Hessiancited by 4
  7. 2024Careful with that Scalpel: Improving Gradient Surgery with an EMAcited by 4
  8. 2023Minimum norm interpolation by perceptra: Explicit regularization and implicit biascited by 3
  9. 2024Deep Loss Convexification for Learning Iterative Modelscited by 3
  10. 2025Flat Channels to Infinity in Neural Loss Landscapescited by 3
  11. 2022FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networkscited by 2
  12. 2023Hierarchical Weight Averaging for Deep Neural Networkscited by 2
  13. 2023Global L2 minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learningcited by 2
  14. 2024On the Convergence of Gradient Descent for Large Learning Ratescited by 2
  15. 2024Enhancing material property prediction with ensemble deep graph convolutional networkscited by 2
  16. 2024Gradient flow in parameter space is equivalent to linear interpolation in output spacecited by 2
  17. 2026Visualizing the loss landscapes of physics-informed neural networkscited by 1
  18. 2023SING: A Plug-and-Play DNN Learning Techniquecited by 0
  19. 2023A simple connection from loss flatness to compressed neural representationscited by 0
  20. 2024Insights into the Lottery Ticket Hypothesis and Iterative Magnitude Pruningcited by 0
  21. 2024Manifold Metric: A Loss Landscape Approach for Predicting Model Performancecited by 0
  22. 2024Dynamical loss functions shape landscape topography and improve learning in artificial neural networkscited by 0
  23. 2025Generalization of the Gibbs algorithm with high probability at low temperaturescited by 0
  24. 2025Architecture independent generalization bounds for overparametrized deep ReLU networkscited by 0
  25. 2025Noise-robust Contrastive Learning for Critical Transition Detection in Dynamical Systemscited by 0
  26. 2026Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memorycited by 0