mlsanityData to 14 Oct 2026

Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data

arXiv 2511.18661 ↗First posted 24 Nov 2025stat.ML cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Scaling laws describe how learning performance improves with data, compute, or training time, and have become a central theme in modern deep learning. We study this phenomenon in a canonical nonlinear model: phase retrieval with anisotropic Gaussian inputs whose covariance spectrum follows a power law. Unlike the isotropic case, where dynamics collapse to a two-dimensional system, anisotropy yields a qualitatively new regime in which an infinite hierarchy of coupled equations governs the evolution of the summary statistics. We develop a tractable reduction that reveals a three-phase trajectory: (i) fast escape from low alignment, (ii) slow convergence of the summary statistics, and (iii) spectral-tail learning in low-variance directions. From this decomposition, we derive explicit scaling laws for the mean-squared error, showing how spectral decay dictates convergence times and error curves. Experiments confirm the predicted phases and exponents. These results provide the first rigorous characterization of scaling laws in nonlinear regression with anisotropic data, highlighting how anisotropy reshapes learning dynamics.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

39 of 52 references.

References · 39

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  4. 2021Explaining Neural Scaling Lawscited by 363
  5. 2016Generalization Properties of Learning with Random Featurescited by 164
  6. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  7. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
  8. 2014Phase Retrieval via Wirtinger Flow: Theory and Algorithmscited by 125
  9. 2022A Solvable Model of Neural Scaling Lawscited by 116
  10. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  11. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  12. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
27 more
  1. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 70
  2. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  3. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  4. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
  5. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 52
  6. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  7. 2022Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networkscited by 50
  8. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  9. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 43
  10. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  11. 2019Online Stochastic Gradient Descent with Arbitrary Initialization Solves Non-smooth, Non-convex Phase Retrievalcited by 34
  12. 2018Learning with SGD and Random Featurescited by 33
  13. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 32
  14. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 26
  15. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 25
  16. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 24
  17. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  18. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  19. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 20
  20. 2017The nonsmooth landscape of phase retrievalcited by 16
  21. 2017Phase Retrieval via Randomized Kaczmarz: Theoretical Guaranteescited by 15
  22. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 15
  23. 2025Survey on Algorithms for multi-index modelscited by 15
  24. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  25. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 10
  26. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  27. 2025Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descentcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Optimal scaling laws in learning hierarchical multi-index modelscited by 9
  2. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 6
  3. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0