mlsanityData to 7 Oct 2026

Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape

arXiv 2305.19510 ↗First posted 31 May 2023cs.LG math.CO stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

We study the loss landscape of both shallow and deep, mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank Jacobian to many regions having deficient rank depending on the amount of overparameterization.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 22 of its 49 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

22 of 49 references.

References · 22

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  6. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  7. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  8. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  9. 2018Tropical Geometry of Deep Neural Networkscited by 92
  10. 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 76
  11. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  12. 2019On Connected Sublevel Sets in Deep Learningcited by 51
10 more
  1. 2021Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sumscited by 37
  2. 2020The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutionscited by 30
  3. 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 29
  4. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  5. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  6. 2019Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theorycited by 21
  7. 2015On the Quality of the Initial Basin in Overspecified Neural Networkscited by 20
  8. 2022Functional dimension of feedforward ReLU neural networkscited by 20
  9. 2020Piecewise linear activations substantially shape the loss surfaces of neural networkscited by 19
  10. 2017The Multilinear Structure of ReLU Networkscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 4; 2026: 6.

16 papers.

Sort:

  1. 2024The Real Tropical Geometry of Neural Networkscited by 23
  2. 2024Continual Learning with Weight Interpolationcited by 13
  3. 2024Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimensioncited by 13
  4. 2026Most ReLU Networks Admit Identifiable Parameterscited by 5
  5. 2025Understanding Learning Invariance in Deep Linear Networkscited by 3
  6. 2025Constraining the outputs of ReLU neural networkscited by 3
  7. 2024Gradient flow in parameter space is equivalent to linear interpolation in output spacecited by 2
  8. 2024A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Featurescited by 1
  9. 2023On non-approximability of zero loss global mathcal L2 minimizers by gradient descent in Deep Learningcited by 0
  10. 2025Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learningcited by 0
  11. 2025Favorability of Loss Landscape with Weight Decay Requires Both Large Overparametrization and Initializationcited by 0
  12. 2026A Dual Certificate Approach to Sparsity in Infinite-Width Shallow Neural Networkscited by 0
4 more
  1. 2026Neural Network Quantization by Learning Low-Loss Subspacescited by 0
  2. 2026Effects of width-dependent model hyperparameters and ℓ2-regularization on the loss landscape of two-layer ReLU networkscited by 0
  3. 2026Cautious optimism for deep parameterized quantum circuitscited by 0
  4. 2026Dimension-Free Rank Lifting from Random Hyperplane Arrangementscited by 0