mlsanityData to 7 Oct 2026

Optimal scaling laws in learning hierarchical multi-index models

arXiv 2602.05846 ↗First posted 5 Feb 2026stat.ML cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In this work, we provide a sharp theory of scaling laws for two-layer neural networks trained on a class of hierarchical multi-index targets, in a genuinely representation-limited regime. We derive exact information-theoretic scaling laws for subspace recovery and prediction error, revealing how the hierarchical features of the target are sequentially learned through a cascade of phase transitions. We further show that these optimal rates are achieved by a simple, target-agnostic spectral estimator, which can be interpreted as the small learning-rate limit of gradient descent on the first-layer weights. Once an adapted representation is identified, the readout can be learned statistically optimally, using an efficient procedure. As a consequence, we provide a unified and rigorous explanation of scaling laws, plateau phenomena, and spectral structure in shallow neural networks trained on such hierarchical targets.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 51 of its 72 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

51 of 72 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  7. 2021Explaining Neural Scaling Lawscited by 353
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  10. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  11. 2016Generalization Properties of Learning with Random Featurescited by 163
  12. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
38 more
  1. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  2. 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 115
  3. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  4. 2023A Theory for Emergence of Complex Skills in Language Modelscited by 111
  5. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  6. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
  7. 2022A Solvable Model of Neural Scaling Lawscited by 101
  8. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  9. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  10. 2022SGD with Large Step Sizes Learns Sparse Featurescited by 66
  11. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  12. 2018The committee machine: Computational to statistical gaps in learning a two-layers neural networkcited by 58
  13. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
  14. 2021A unifying tutorial on Approximate Message Passingcited by 49
  15. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  16. 2017Fundamental Limits of Weak Recovery with Applications to Phase Retrievalcited by 39
  17. 2024Computational-Statistical Gaps in Gaussian Single-Index Modelscited by 39
  18. 2021Graph-based Approximate Message Passing Iterationscited by 35
  19. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  20. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 29
  21. 2017Phase Transitions of Spectral Initialization for High-Dimensional Nonconvex Estimationcited by 26
  22. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  23. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  24. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 23
  25. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 19
  26. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  27. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  28. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
  29. 2025Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applicationscited by 11
  30. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  31. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  32. 2024Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectracited by 9
  33. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 8
  34. 2024The Role of the Time-Dependent Hessian in High-Dimensional Optimizationcited by 7
  35. 2025Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recoverycited by 6
  36. 2025Optimal Spectral Transitions in High-Dimensional Multi-Index Modelscited by 6
  37. 2026High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networkscited by 5
  38. 2025Neural Networks Learn Generic Multi-Index Models Near Information-Theoretic Limitcited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Learning Sparse Compositional Functions with Norm-Constrained Neural Networkscited by 2
  2. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 1
  3. 2026Learn from your own latents and not from tokens: A sample-complexity theorycited by 1
  4. 2026Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networkscited by 0
  5. 2026How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscited by 0
  6. 2026Coupled Scaling: A Representational Accessibility Framework for Neural Scaling Lawscited by 0