mlsanityData to 7 Oct 2026

The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks

arXiv 2006.01005 ↗First posted 1 Jun 2020cs.LG stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

We study the effects of mild over-parameterization on the optimization landscape of a simple ReLU neural network of the form xmapstosumi=1kmax0,witopx, in a well-studied teacher-student setting where the target values are generated by the same architecture, and when directly optimizing over the population squared loss with respect to Gaussian inputs. We prove that while the objective is strongly convex around the global minima when the teacher and student networks possess the same number of neurons, it is not even locally convex after any amount of over-parameterization. Moreover, related desirable properties (e.g., one-point strong convexity and the Polyak-Lojasiewicz condition) also do not hold even locally. On the other hand, we establish that the objective remains one-point strongly convex in most directions (suitably defined), and show an optimization guarantee under this property. For the non-global minima, we prove that adding even just a single neuron will turn a non-global minimum into a saddle point. This holds under some technical conditions which we validate empirically. These results provide a possible explanation for why recovering a global minimum becomes significantly easier when we over-parameterize, even if the amount of over-parameterization is very moderate.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 11; 2023: 9; 2024: 2; 2025: 3; 2026: 4.

33 papers.

Sort:

By area:

  1. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  2. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  3. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  4. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  5. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  6. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  7. 2021Overparameterization of deep ResNet: zero loss and mean-field analysiscited by 24
  8. 2023Layer-wise Linear Mode Connectivitycited by 22
  9. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  10. 2022Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)cited by 18
  11. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  12. 2022Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methodscited by 10
21 more
  1. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  2. 2022Annihilation of Spurious Minima in Two-Layer ReLU Networkscited by 9
  3. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
  4. 2023Should Under-parameterized Student Networks Copy or Average Teacher Weights?cited by 8
  5. 2022Optimization for Amortized Inverse Problemscited by 6
  6. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  7. 2022Early Stage Convergence and Global Convergence of Training Mildly Parameterized Neural Networkscited by 5
  8. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  9. 2023Primal Dual Continual Learning: Balancing Stability and Plasticity through Adaptive Memory Allocationcited by 4
  10. 2023Fast Convergence in Learning Two-Layer Neural Networks with Separable Datacited by 3
  11. 2023FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learningcited by 3
  12. 2025Flat Channels to Infinity in Neural Loss Landscapescited by 3
  13. 2022Do we need Label Regularization to Fine-tune Pre-trained Language Models?cited by 2
  14. 2023Hidden Minima in Two-Layer ReLU Networkscited by 2
  15. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  16. 2025Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLUcited by 1
  17. 2025A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Pointcited by 1
  18. 2026Hidden Gauge Controls Feature Specialization in ReLU Networkscited by 1
  19. 2023Modify Training Directions in Function Space to Reduce Generalization Errorcited by 0
  20. 2024Chasing Better Deep Image Priors between Over- and Under-parameterizationcited by 0
  21. 2026Removing spurious minima for planar features by skip connectionscited by 0