mlsanityData to 7 Oct 2026

Universality of high-dimensional scaling limits of stochastic gradient descent

arXiv 2512.13634 ↗First posted 15 Dec 2025stat.ML cs.LG math.PRcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider statistical tasks in high dimensions whose loss depends on the data only through its projection into a fixed-dimensional subspace spanned by the parameter vectors and certain ground truth vectors. This includes classifying mixture distributions with cross-entropy loss with one and two-layer networks, and learning single and multi-index models with one and two-layer networks. When the data is drawn from an isotropic Gaussian mixture distribution, it is known that the evolution of a finite family of summary statistics under stochastic gradient descent converges to an autonomous ordinary differential equation (ODE), as the dimension and sample size go to ∞ and the step size goes to 0 commensurately. Our main result is that these ODE limits are universal in that this limit is the same whenever the data is drawn from mixtures of arbitrary product distributions whose first two moments match the corresponding Gaussian distribution, provided the initialization and ground truth vectors are coordinate-delocalized. We complement this by proving two corresponding non-universality results. We provide a simple example where the ODE limits are non-universal if the initialization is coordinate aligned. We also show that the stochastic differential equation limits arising as fluctuations of the summary statistics around their ODE's fixed points are not universal.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 15 Dec 2025.

Builds on

32 of 54 references.

References · 32

  1. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  2. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  3. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  4. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  5. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  6. 2022Universality of empirical risk minimizationcited by 73
  7. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  8. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  9. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  10. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 60
  11. 2023On Learning Gaussian Multi-index Models with Gradient Flowcited by 53
  12. 2020Learning a Single Neuron with Gradient Methodscited by 50
20 more
  1. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  2. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  3. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  4. 2024Computational-Statistical Gaps in Gaussian Single-Index Modelscited by 39
  5. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  6. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  7. 2021SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticalitycited by 33
  8. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  9. 2023Universality laws for Gaussian mixtures in generalized linear modelscited by 28
  10. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 25
  11. 2023On Single Index Models beyond Gaussian Datacited by 17
  12. 2020Universality of Approximate Message Passing Algorithmscited by 16
  13. 2017Phase Retrieval via Randomized Kaczmarz: Theoretical Guaranteescited by 15
  14. 2022Learning a Single Neuron for Non-monotonic Activation Functionscited by 13
  15. 2023Spectral alignment of stochastic gradient descent for high-dimensional classification taskscited by 13
  16. 2017Scaling Limit: Exact and Tractable Analysis of Online Learning Algorithms with Applications to Regularized Regression and PCAcited by 11
  17. 2018Online ICA: Understanding Global Dynamics of Nonconvex Optimization via Diffusion Processescited by 6
  18. 2023High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariancecited by 5
  19. 2025Exact Dynamics of Multi-class Stochastic Gradient Descentcited by 1
  20. 2025Limit Theorems for Stochastic Gradient Descent in High-Dimensional Single-Layer Networkscited by 1

Built on by

Not cited yet.