mlsanityData to 7 Oct 2026

Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networks

arXiv 2511.02258 ↗First posted 4 Nov 2025stat.ML cs.LG math.PRcited by 1

Not on the map. Why?

Abstract (arXiv)

This paper studies the high-dimensional scaling limits of online stochastic gradient descent (SGD). Building on the work of Ben Arous, Gheissari, and Jagannath on the effective dynamics of SGD, we study the critical scaling regime of the step size for single-layer networks. Below this regime the effective dynamics are governed by deterministic (ballistic) limits, whereas at the critical scale a correction term emerges that changes the phase diagram. Near the fixed points of these dynamics, we show that the diffusive (SDE) limit of the rescaled correlation is an Ornstein-Uhlenbeck process. More precisely, it is mean-reverting whenever the information exponent is at least three. At information exponent two the drift has no universal sign, and the fixed point may become repelling; we show this explicitly for phase retrieval, where the sign is determined by the step size and the noise level. These results illustrate the limitations of deterministic scaling limits in capturing stochastic fluctuations in high-dimensional learning dynamics.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 30 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

17 of 30 references.

References · 17

  1. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  3. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  4. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
  5. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  6. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  7. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  8. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  9. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  10. 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 94
  11. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  12. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
5 more
  1. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  2. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  3. 2019Normal Approximation for Stochastic Gradient Descent via Non-Asymptotic Rates of Martingale CLTcited by 30
  4. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 25
  5. 2018Diffusion Approximations for Online Principal Component Estimation and Global Convergencecited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Universality of high-dimensional scaling limits of stochastic gradient descentcited by 0