# Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra

arXiv: https://arxiv.org/abs/2410.09005 · HTML: https://mlsanity.com/paper/2410.09005/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 26 of 26)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [Learning curves of generic features maps for realistic datasets with a teacher-student model](https://mlsanity.com/paper/2102.08127/) (2021)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [The Gaussian equivalence of generative models for learning with shallow neural networks](https://mlsanity.com/paper/2006.14709/) (2020)
- [A Theory for Emergence of Complex Skills in Language Models](https://mlsanity.com/paper/2307.15936/) (2023)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018)
- [More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize](https://mlsanity.com/paper/2203.06176/) (2022)
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Model Collapse Demystified: The Case of Regression](https://mlsanity.com/paper/2402.07712/) (2024)
- [How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model](https://mlsanity.com/paper/2307.02129/) (2023)
- [4+3 Phases of Compute-Optimal Neural Scaling Laws](https://mlsanity.com/paper/2405.15074/) (2024)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [How Feature Learning Can Improve Neural Scaling Laws](https://mlsanity.com/paper/2409.17858/) (2024)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Towards a theory of how the structure of language is acquired by deep neural networks](https://mlsanity.com/paper/2406.00048/) (2024)
- [Scaling laws for learning with real and surrogate data](https://mlsanity.com/paper/2402.04376/) (2024)
- [Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysis](https://mlsanity.com/paper/2001.03371/) (2020)
- [An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem](https://mlsanity.com/paper/2404.17563/) (2024)

## Built on by (most cited first, 9 of 9)

- [Universal One-third Time Scaling in Learning Peaked Distributions](https://mlsanity.com/paper/2602.03685/) (2026)
- [Optimal scaling laws in learning hierarchical multi-index models](https://mlsanity.com/paper/2602.05846/) (2026)
- [Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrieval](https://mlsanity.com/paper/2601.22652/) (2026)
- [Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients](https://mlsanity.com/paper/2606.25008/) (2026)
- [Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data](https://mlsanity.com/paper/2511.18661/) (2025)
- [A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification](https://mlsanity.com/paper/2605.22341/) (2026)
- [Asymmetric Scaling Laws from Sparse Features](https://mlsanity.com/paper/2605.23591/) (2026)
- [Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail](https://mlsanity.com/paper/2605.31244/) (2026)
- [Emergent One-Third Scaling Law as Attention Tries to Concentrate](https://mlsanity.com/paper/2609.32100/) (2026)

## Abstract (arXiv)

Neural scaling laws describe how the performance of deep neural networks scales with key factors such as training data size, model complexity, and training time, often following power-law behaviors over multiple orders of magnitude. Despite their empirical observation, the theoretical understanding of these scaling laws remains limited. In this work, we employ techniques from statistical mechanics to analyze one-pass stochastic gradient descent within a student-teacher framework, where both the student and teacher are two-layer neural networks. Our study primarily focuses on the generalization error and its behavior in response to data covariance matrices that exhibit power-law spectra. For linear activation functions, we derive analytical expressions for the generalization error, exploring different learning regimes and identifying conditions under which power-law scaling emerges. Additionally, we extend our analysis to non-linear activation functions in the feature learning regime, investigating how power-law spectra in the data covariance matrix impact learning dynamics. Importantly, we find that the length of the symmetric plateau depends on the number of distinct eigenvalues of the data covariance matrix and the number of hidden units, demonstrating how these plateaus behave under various configurations. In addition, our results reveal a transition from exponential to power-law convergence in the specialized phase when the data covariance matrix possesses a power-law spectrum. This work contributes to the theoretical understanding of neural scaling laws and provides insights into optimizing learning performance in practical scenarios involving complex data structures.