mlsanityData to 14 Oct 2026

Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime

arXiv 2509.24882 ↗First posted 29 Sep 2025cs.LG cond-mat.dis-nn cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime. Leveraging connections with matrix compressed sensing and LASSO, we derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay. This analysis uncovers crossovers between distinct scaling regimes and plateau behaviors, mirroring phenomena widely reported in the empirical neural scaling literature. Furthermore, we establish a precise link between these regimes and the spectral properties of the trained network weights, which we characterize in detail. As a consequence, we provide a theoretical validation of recent empirical observations connecting the emergence of power-law tails in the weight spectrum with network generalization performance, yielding an interpretation from first principles.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 51 of its 75 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

51 of 75 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  4. 2019Benign Overfitting in Linear Regressioncited by 549
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  6. 2021Explaining Neural Scaling Lawscited by 363
  7. 2017Implicit Regularization in Matrix Factorizationcited by 322
  8. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  9. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  10. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  11. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  12. 2022A Solvable Model of Neural Scaling Lawscited by 116
38 more
  1. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  2. 2024A Dynamical Model of Neural Scaling Lawscited by 111
  3. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  4. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  5. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  6. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  7. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 57
  8. 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
  9. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 53
  10. 2022Dimension free ridge regressioncited by 51
  11. 2020Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functionscited by 50
  12. 2021A unifying tutorial on Approximate Message Passingcited by 50
  13. 2024Scaling and renormalization in high-dimensional regressioncited by 50
  14. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  15. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  16. 2021Graph-based Approximate Message Passing Iterationscited by 35
  17. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 32
  18. 2022Spectral Evolution and Invariance in Linear-width Neural Networkscited by 30
  19. 2024A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimatorcited by 28
  20. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 25
  21. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 24
  22. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  23. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  24. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  25. 2016Finite Sample Analysis of Approximate Message Passing Algorithmscited by 20
  26. 2022A Non-Asymptotic Framework for Approximate Message Passing in Spiked Modelscited by 20
  27. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 20
  28. 2021Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metricscited by 19
  29. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 16
  30. 2023Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGDcited by 15
  31. 2025Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applicationscited by 11
  32. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  33. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 10
  34. 2025From SGD to Spectra: A Theory of Neural Network Weight Dynamicscited by 9
  35. 2023Fitting an ellipsoid to random points: predictions using the replica methodcited by 7
  36. 2023Exact threshold for approximate ellipsoid fitting of random pointscited by 7
  37. 2024Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspectivecited by 4
  38. 2024Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinementcited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 18.

20 papers.

Sort:

  1. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  2. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 12
  3. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 10
  4. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 10
  5. 2026Optimal scaling laws in learning hierarchical multi-index modelscited by 9
  6. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 6
  7. 2026Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercited by 6
  8. 2026Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficientscited by 5
  9. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  10. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 2
  11. 2026Spectral phase transitions and trainability in neural network learning dynamicscited by 2
  12. 2026Asymmetric Scaling Laws from Sparse Featurescited by 1
8 more
  1. 2026Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regimecited by 1
  2. 2026Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networkscited by 0
  3. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0
  4. 2026How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscited by 0
  5. 2026Dual-Manifold Geometry Guided Representation Learning: Adaptive Coupling between Kernel and Data Spacescited by 0
  6. 2026Coupled Scaling: A Representational Accessibility Framework for Neural Scaling Lawscited by 0
  7. 2026High-Dimensional Learning Dynamics of Attention-Indexed Modelscited by 0
  8. 2026Emergent One-Third Scaling Law as Attention Tries to Concentratecited by 0