mlsanityData to 7 Oct 2026

Learning time-scales in two-layers neural networks

arXiv 2303.00055 ↗First posted 28 Feb 2023cs.LG math.OC stat.MLcited by 57

Not on the map. Why?

Abstract (arXiv)

Gradient-based learning in multi-layer neural networks displays a number of striking features. In particular, the decrease rate of empirical risk is non-monotone even after averaging over large batches. Long plateaus in which one observes barely any progress alternate with intervals of rapid decrease. These successive phases of learning often take place on very different time scales. Finally, models learnt in an early phase are typically `simpler' or `easier to learn' although in a way that is difficult to formalize. Although theoretical explanations of these phenomena have been put forward, each of them captures at best certain specific regimes. In this paper, we study the gradient flow dynamics of a wide two-layer neural network in high-dimension, when data are distributed according to a single-index model (i.e., the target function depends on a one-dimensional projection of the covariates). Based on a mixture of new rigorous results, non-rigorous mathematical derivations, and numerical simulations, we propose a scenario for the learning dynamics in this setting. In particular, the proposed evolution exhibits separation of timescales and intermittency. These behaviors arise naturally because the population gradient flow can be recast as a singularly perturbed dynamical system.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

28 of 40 references.

References · 28

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2017A Closer Look at Memorization in Deep Networkscited by 936
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2021Deep learning: a statistical viewpointcited by 239
  8. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  9. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  10. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  11. 2020Feature Learning in Infinite-Width Neural Networkscited by 182
  12. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
16 more
  1. 2019Linearized two-layers neural networks in high dimensioncited by 155
  2. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  3. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  4. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  5. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  6. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  7. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  8. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 105
  9. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  10. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  11. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  12. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  13. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  14. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 22
  15. 2021The Three Stages of Learning Dynamics in High-Dimensional Kernel Methodscited by 21
  16. 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 16; 2025: 17; 2026: 13.

57 papers.

Sort:

By area:

  1. 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
  2. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  3. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  4. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  5. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  6. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  7. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  8. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  9. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  10. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 24
  11. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  12. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
38 more
  1. 2023Six Lectures on Linearized Neural Networkscited by 19
  2. 2023On Single Index Models beyond Gaussian Datacited by 17
  3. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
  4. 2024Attention layers provably solve single-location regressioncited by 15
  5. 2025Survey on Algorithms for multi-index modelscited by 15
  6. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 14
  7. 2024Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamicscited by 14
  8. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  9. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  10. 2024Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinationscited by 13
  11. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  12. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  13. 2023Phase transitions in the mini-batch size for sparse and dense two-layer neural networkscited by 9
  14. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  15. 2023Should Under-parameterized Student Networks Copy or Average Teacher Weights?cited by 8
  16. 2024Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergencecited by 8
  17. 2025Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributionscited by 8
  18. 2025Optimal Spectral Transitions in High-Dimensional Multi-Index Modelscited by 6
  19. 2024Task Diversity Shortens the ICL Plateaucited by 5
  20. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 5
  21. 2026A theory of learning data statistics in diffusion models, from easy to hardcited by 5
  22. 2024Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depthcited by 4
  23. 2024Failures and Successes of Cross-Validation for Early-Stopped Gradient Descentcited by 4
  24. 2025Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regimecited by 4
  25. 2025Bayes optimal learning of attention-indexed modelscited by 3
  26. 2025Test time training enhances in-context learning of nonlinear functionscited by 3
  27. 2025Closed-Form Last Layer Optimizationcited by 3
  28. 2026High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Modelcited by 3
  29. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 2
  30. 2025Convergence of two-timescale gradient descent ascent dynamics: finite-dimensional and mean-field perspectivescited by 2
  31. 2025From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcited by 2
  32. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  33. 2025Joint Learning in the Gaussian Single Index Modelcited by 1
  34. 2026Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Paritycited by 1
  35. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 1
  36. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  37. 2025Gaussian mixture layers for neural networkscited by 0
  38. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0

50 most cited of 57 All 57 (JSON)