# Emergence and scaling laws in SGD learning of shallow neural networks

arXiv: https://arxiv.org/abs/2504.19983 · HTML: https://mlsanity.com/paper/2504.19983/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2025.

## Built on by (most cited first, 30 of 32)

- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [Deriving Neural Scaling Laws from the statistics of natural language](https://mlsanity.com/paper/2602.07488/) (2026)
- [Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime](https://mlsanity.com/paper/2509.24882/) (2025)
- [Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks](https://mlsanity.com/paper/2506.06489/) (2025)
- [On the origin of neural scaling laws: from random graphs to natural language](https://mlsanity.com/paper/2601.10684/) (2026)
- [Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory](https://mlsanity.com/paper/2603.26554/) (2026)
- [Improved Scaling Laws in Linear Regression via Data Reuse](https://mlsanity.com/paper/2506.08415/) (2025)
- [Kernel ridge regression under power-law data: spectrum and generalization](https://mlsanity.com/paper/2510.04780/) (2025)
- [Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention](https://mlsanity.com/paper/2605.29548/) (2026)
- [The Generative Leap: Sharp Sample Complexity for Efficiently Learning Gaussian Multi-Index Models](https://mlsanity.com/paper/2506.05500/) (2025)
- [Optimal scaling laws in learning hierarchical multi-index models](https://mlsanity.com/paper/2602.05846/) (2026)
- [Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws](https://mlsanity.com/paper/2509.24914/) (2025)
- [Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining](https://mlsanity.com/paper/2601.19756/) (2026)
- [Power-Law Spectrum of the Random Feature Model](https://mlsanity.com/paper/2603.14578/) (2026)
- [Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data](https://mlsanity.com/paper/2511.18661/) (2025)
- [Generalization performance of narrow one-hidden layer networks in the teacher-student setting](https://mlsanity.com/paper/2507.00629/) (2025)
- [Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts](https://mlsanity.com/paper/2510.07205/) (2025)
- [From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGD](https://mlsanity.com/paper/2510.21020/) (2025)
- [A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models](https://mlsanity.com/paper/2603.17896/) (2026)
- [Effective Frontiers: A Unification of Neural Scaling Laws](https://mlsanity.com/paper/2602.02593/) (2026)
- [The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledge](https://mlsanity.com/paper/2605.12908/) (2026)
- [Asymmetric Scaling Laws from Sparse Features](https://mlsanity.com/paper/2605.23591/) (2026)
- [Thinned Mean Field Langevin Dynamics](https://mlsanity.com/paper/2605.28589/) (2026)
- [When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks](https://mlsanity.com/paper/2606.04476/) (2026)
- [Predicting Large Model Test Losses with a Noisy Quadratic System](https://mlsanity.com/paper/2605.09154/) (2026)
- [Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks](https://mlsanity.com/paper/2605.10395/) (2026)
- [Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model](https://mlsanity.com/paper/2605.14567/) (2026)
- [A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights](https://mlsanity.com/paper/2605.16913/) (2026)
- [Convex Basins in Single-Index Model Loss Landscapes: Applications to Robust Recovery under Strong Adversarial Corruption](https://mlsanity.com/paper/2605.29497/) (2026)
- [Theory of learning of high-dimensional controlled non-linear dynamical systems (I): models and methods](https://mlsanity.com/paper/2606.07247/) (2026)

## Abstract (arXiv)

We study the complexity of online stochastic gradient descent (SGD) for learning a two-layer neural network with P neurons on isotropic Gaussian data: f_*(boldsymbolx) = sump=1P ap· σ(langleboldsymbolx,boldsymbolvp^*rangle), boldsymbolx ~ N(0,boldsymbolId), where the activation σ:ℝ→ℝ is an even function with information exponent k_*>2 (defined as the lowest degree in the Hermite expansion), boldsymbolv^*pp∈[P]subset ℝd are orthonormal signal directions, and the non-negative second-layer coefficients satisfy sump ap2=1. We focus on the challenging ``extensive-width'' regime Pgg 1 and permit diverging condition number in the second-layer, covering as a special case the power-law scaling apasymp p-β where β∈ℝ≥ 0. We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective, and explicitly identify sharp transition times to recover each signal direction. In the power-law setting, we characterize scaling law exponents for the MSE loss with respect to the number of training samples and SGD steps, as well as the number of parameters in the student neural network. Our analysis entails that while the learning of individual teacher neurons exhibits abrupt transitions, the juxtaposition of Pgg 1 emergent learning curves at different timescales leads to a smooth scaling law in the cumulative objective.