mlsanityData to 14 Oct 2026

Singular perturbations and hierarchical learning in two-layer neural networks

arXiv 2607.10869 ↗First posted 12 Jul 2026cs.LG math.DS math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are optimized jointly, with a perturbative parameter tuning the relative training speed between the first and second layer. This setting was considered by Berthier, Montanari and Zhou in citeberthier2024learning, who conjectured a hierarchical learning scenario with explicit timescales as the second layer is trained faster than the first. In this paper, we prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales, at sharp explicit thresholds. We then analyze the onset of learning of the quadratic component and show that the components learned at earlier stages continue to influence the dynamics in an essential way. Our proof is based on quantitative approximation results for singularly perturbed flows evolving near a manifold defined by integral constraints. At a phenomenological level, we also show that the empirical measure of the weights displays singular behaviour when reaching the quadratic component of the hidden link, with a small fraction of neurons growing significantly while the remaining ones rearrange to preserve the components already learned.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 12 Jul 2026.

Builds on

43 of 54 references.

References · 43

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  2. 2017A Closer Look at Memorization in Deep Networkscited by 942
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  4. 2018On Lazy Training in Differentiable Programmingcited by 675
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  6. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  7. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  8. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 331
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  10. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  11. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  12. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
31 more
  1. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  2. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  3. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  4. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  5. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  6. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  7. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  8. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
  9. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  10. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  11. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 85
  12. 2020Learning Parities with Neural Networkscited by 78
  13. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 70
  14. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  15. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  16. 2023Learning time-scales in two-layers neural networkscited by 57
  17. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  18. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  19. 2020Learning a Single Neuron with Gradient Methodscited by 50
  20. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  21. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  22. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 43
  23. 2020On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regimecited by 39
  24. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 39
  25. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  26. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  27. 2023Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Timecited by 20
  28. 2023Leveraging the two timescale regime to demonstrate convergence of neural networkscited by 19
  29. 2022On the symmetries in the dynamics of wide two-layer neural networkscited by 8
  30. 2026Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networkscited by 3
  31. 2025Joint Learning in the Gaussian Single Index Modelcited by 1

Built on by

Not cited yet.