# Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime

arXiv: https://arxiv.org/abs/2509.24882 · HTML: https://mlsanity.com/paper/2509.24882/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 75 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 30 of 51)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning](https://mlsanity.com/paper/1810.01075/) (2018)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Learning curves of generic features maps for realistic datasets with a teacher-student model](https://mlsanity.com/paper/2102.08127/) (2021)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data](https://mlsanity.com/paper/2002.06716/) (2020)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Traditional and Heavy-Tailed Self Regularization in Neural Network Models](https://mlsanity.com/paper/1901.08276/) (2019)
- [Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime](https://mlsanity.com/paper/2105.15004/) (2021)
- [How Two-Layer Neural Networks Learn, One (Giant) Step at a Time](https://mlsanity.com/paper/2305.18270/) (2023)
- [4+3 Phases of Compute-Optimal Neural Scaling Laws](https://mlsanity.com/paper/2405.15074/) (2024)
- [Saddle-to-Saddle Dynamics in Diagonal Linear Networks](https://mlsanity.com/paper/2304.00488/) (2023)
- [How Feature Learning Can Improve Neural Scaling Laws](https://mlsanity.com/paper/2409.17858/) (2024)
- [Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification](https://mlsanity.com/paper/2004.09031/) (2020)
- [Dimension free ridge regression](https://mlsanity.com/paper/2210.08571/) (2022)
- [Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions](https://mlsanity.com/paper/2006.15459/) (2020)
- [A unifying tutorial on Approximate Message Passing](https://mlsanity.com/paper/2105.02180/) (2021)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks](https://mlsanity.com/paper/2310.07891/) (2023)
- [Asymptotics of feature learning in two-layer networks after one gradient-step](https://mlsanity.com/paper/2402.04980/) (2024)
- [Graph-based Approximate Message Passing Iterations](https://mlsanity.com/paper/2109.11905/) (2021)
- [Emergence and scaling laws in SGD learning of shallow neural networks](https://mlsanity.com/paper/2504.19983/) (2025)
- [Spectral Evolution and Invariance in Linear-width Neural Networks](https://mlsanity.com/paper/2211.06506/) (2022)

## Built on by (most cited first, 20 of 20)

- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [HTMuon: Improving Muon via Heavy-Tailed Spectral Correction](https://mlsanity.com/paper/2603.10067/) (2026)
- [Kernel ridge regression under power-law data: spectrum and generalization](https://mlsanity.com/paper/2510.04780/) (2025)
- [Universal One-third Time Scaling in Learning Peaked Distributions](https://mlsanity.com/paper/2602.03685/) (2026)
- [Optimal scaling laws in learning hierarchical multi-index models](https://mlsanity.com/paper/2602.05846/) (2026)
- [Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval](https://mlsanity.com/paper/2601.22652/) (2026)
- [Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfer](https://mlsanity.com/paper/2605.07870/) (2026)
- [Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients](https://mlsanity.com/paper/2606.25008/) (2026)
- [Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data](https://mlsanity.com/paper/2511.18661/) (2025)
- [A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Models](https://mlsanity.com/paper/2603.17896/) (2026)
- [Spectral phase transitions and trainability in neural network learning dynamics](https://mlsanity.com/paper/2606.28486/) (2026)
- [Asymmetric Scaling Laws from Sparse Features](https://mlsanity.com/paper/2605.23591/) (2026)
- [Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime](https://mlsanity.com/paper/2605.29684/) (2026)
- [Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks](https://mlsanity.com/paper/2605.10395/) (2026)
- [A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights](https://mlsanity.com/paper/2605.16913/) (2026)
- [How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks](https://mlsanity.com/paper/2606.28242/) (2026)
- [Dual-Manifold Geometry Guided Representation Learning: Adaptive Coupling between Kernel and Data Spaces](https://mlsanity.com/paper/2608.12737/) (2026)
- [Coupled Scaling: A Representational Accessibility Framework for Neural Scaling Laws](https://mlsanity.com/paper/2609.03533/) (2026)
- [High-Dimensional Learning Dynamics of Attention-Indexed Models](https://mlsanity.com/paper/2609.03858/) (2026)
- [Emergent One-Third Scaling Law as Attention Tries to Concentrate](https://mlsanity.com/paper/2609.32100/) (2026)

## Abstract (arXiv)

Neural scaling laws underlie many of the recent advances in deep learning, yet their theoretical understanding remains largely confined to linear models. In this work, we present a systematic analysis of scaling laws for quadratic and diagonal neural networks in the feature learning regime. Leveraging connections with matrix compressed sensing and LASSO, we derive a detailed phase diagram for the scaling exponents of the excess risk as a function of sample complexity and weight decay. This analysis uncovers crossovers between distinct scaling regimes and plateau behaviors, mirroring phenomena widely reported in the empirical neural scaling literature. Furthermore, we establish a precise link between these regimes and the spectral properties of the trained network weights, which we characterize in detail. As a consequence, we provide a theoretical validation of recent empirical observations connecting the emergence of power-law tails in the weight spectrum with network generalization performance, yielding an interpretation from first principles.