mlsanityData to 7 Oct 2026

Leveraging the two timescale regime to demonstrate convergence of neural networks

arXiv 2304.09576 ↗First posted 19 Apr 2023math.OC cs.LG stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

We study the training dynamics of shallow neural networks, in a two-timescale regime in which the stepsizes for the inner layer are much smaller than those for the outer layer. In this regime, we prove convergence of the gradient flow to a global optimum of the non-convex optimization problem in a simple univariate setting. The number of neurons need not be asymptotically large for our result to hold, distinguishing our result from popular recent approaches such as the neural tangent kernel or mean-field regimes. Experimental illustration is provided, showing that the stochastic gradient descent behaves according to our description of the gradient flow and thus converges to a global optimum in the two-timescale regime, but can fail outside of this regime.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 14 of its 33 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

14 of 33 references.

References · 14

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  7. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  8. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  9. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 141
  10. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  11. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  12. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
2 more
  1. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  2. 2015Layer-Specific Adaptive Learning Rates for Deep Networkscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 6; 2026: 7.

18 papers.

Sort:

By area:

  1. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  2. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  3. 2024Implicit Diffusion: Efficient Optimization through Stochastic Samplingcited by 22
  4. 2024Attention layers provably solve single-location regressioncited by 15
  5. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  6. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  7. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  8. 2025Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regimecited by 4
  9. 2025Closed-Form Last Layer Optimizationcited by 3
  10. 2025From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcited by 2
  11. 2026Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Paritycited by 1
  12. 2026Rethinking Neural Network Learning Rates: A Stackelberg Perspectivecited by 1
6 more
  1. 2025Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learningcited by 0
  2. 2026Discovering Scaling Exponents with Physics-Informed M"untz-Sz'asz Networkscited by 0
  3. 2026Attention-based PCAcited by 0
  4. 2026Singular perturbations and hierarchical learning in two-layer neural networkscited by 0
  5. 2026Teacher Geometry Shapes Learnability in Teacher-Student Networkscited by 0
  6. 2026Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networkscited by 0