mlsanityData to 7 Oct 2026

Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networks

arXiv 2604.03068 ↗First posted 3 Apr 2026cond-mat.dis-nn cond-mat.stat-mech stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We analyze the one-pass stochastic gradient descent dynamics of a two-layer neural network with quadratic activations in a teacher--student framework. In the high-dimensional regime, where the input dimension N and the number of samples M diverge at fixed ratio α = M/N, and for finite hidden widths (p,p^*) of the student and teacher, respectively, we study the low-dimensional ordinary differential equations that govern the evolution of the student--teacher and student--student overlap matrices. We show that overparameterization (p>p^*) only modestly accelerates escape from a plateau of poor generalization by modifying the prefactor of the exponential decay of the loss. We then examine how unconstrained weight norms introduce a continuous rotational symmetry that results in a nontrivial manifold of zero-loss solutions for p>1. From this manifold the dynamics consistently selects the closest solution to the random initialization, as enforced by a conserved quantity in the ODEs governing the evolution of the overlaps. Finally, a Hessian analysis of the population-loss landscape confirms that the plateau and the solution manifold correspond to saddles with at least one negative eigenvalue and to marginal minima in the population-loss geometry, respectively.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 54 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

42 of 54 references.

References · 42

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  5. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 283
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  9. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  10. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  11. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  12. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
30 more
  1. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  2. 2014Phase Retrieval via Wirtinger Flow: Theory and Algorithmscited by 123
  3. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  4. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  5. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  6. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  7. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  8. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  9. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  10. 2013Phase Retrieval using Alternating Minimizationcited by 58
  11. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  12. 2023Learning time-scales in two-layers neural networkscited by 57
  13. 2015Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systemscited by 50
  14. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  15. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  16. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  17. 2021Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networkscited by 47
  18. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
  19. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  20. 2024Computational-Statistical Gaps in Gaussian Single-Index Modelscited by 39
  21. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  22. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  23. 2025Symmetry in Neural Network Parameter Spacescited by 31
  24. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 25
  25. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  26. 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networkscited by 21
  27. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  28. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  29. 2019Generalisation dynamics of online learning in over-parameterised neural networkscited by 9
  30. 2016Solving Large-scale Systems of Random Quadratic Equations via Stochastic Truncated Amplitude Flowcited by 1

Built on by

Not cited yet.