mlsanityData to 14 Oct 2026

Annihilation of Spurious Minima in Two-Layer ReLU Networks

arXiv 2210.06088 ↗First posted 12 Oct 2022cs.LG math.OCcited by 9

Not on the map. Why?

Abstract (arXiv)

We study the optimization problem associated with fitting two-layer ReLU neural networks with respect to the squared loss, where labels are generated by a target network. Use is made of the rich symmetry structure to develop a novel set of tools for studying the mechanism by which over-parameterization annihilates spurious minima. Sharp analytic estimates are obtained for the loss and the Hessian spectrum at different minima, and it is proved that adding neurons can turn symmetric spurious minima into saddles; minima of lesser symmetry require more neurons. Using Cauchy's interlacing theorem, we prove the existence of descent directions in certain subspaces arising from the symmetry structure of the loss function. This analytic approach uses techniques, new to the field, from algebraic geometry, representation theory and symmetry breaking, and confirms rigorously the effectiveness of over-parameterization in making the associated loss landscape accessible to gradient-based methods. For a fixed number of neurons and inputs, the spectral results remain true under symmetry breaking perturbation of the target.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

32 of 50 references.

References · 32

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  5. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  6. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  7. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  8. 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
  9. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  10. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  11. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  12. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
20 more
  1. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  2. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  3. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 81
  4. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  5. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  6. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  7. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  8. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  9. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  10. 2020The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networkscited by 33
  11. 2016Distribution-Specific Hardness of Learning Neural Networkscited by 27
  12. 2020Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetrycited by 17
  13. 2021On Learnability via Gradient Method for Two-Layer ReLU Neural Networks in Teacher-Student Settingcited by 16
  14. 2021Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry IIcited by 16
  15. 2019Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activationcited by 14
  16. 2019On the Principle of Least Symmetry Breaking in Shallow ReLU Modelscited by 10
  17. 2020Symmetry & critical points for a model shallow neural networkcited by 10
  18. 2021Symmetry Breaking in Symmetric Tensor Decompositioncited by 10
  19. 2019Generalisation dynamics of online learning in over-parameterised neural networkscited by 9
  20. 2021Equivariant bifurcation, quadratic equivariants, and symmetry breaking for the standard representation of Sncited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 2; 2026: 1.

9 papers.

  1. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  2. 2023Symmetry & Critical Points for Symmetric Tensor Decomposition Problemscited by 3
  3. 2023Hidden Minima in Two-Layer ReLU Networkscited by 2
  4. 2024Symmetry & Critical Pointscited by 2
  5. 2024Probability Distribution Learning and Its Application in Deep Learningcited by 1
  6. 2025Nonlinear Dynamics In Optimization Landscape of Shallow Neural Networks with Tunable Leaky ReLUcited by 1
  7. 2024Error Bounds of Supervised Classification from Information-Theoretic Perspectivecited by 0
  8. 2025Towards Understanding the Optimization Mechanisms in Deep Learningcited by 0
  9. 2026Removing spurious minima for planar features by skip connectionscited by 0