# Power-Law Spectrum of the Random Feature Model

arXiv: https://arxiv.org/abs/2603.14578 · HTML: https://mlsanity.com/paper/2603.14578/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 15 Mar 2026.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Deep learning: a statistical viewpoint](https://mlsanity.com/paper/2103.09177/) (2021)
- [Benign overfitting in ridge regression](https://mlsanity.com/paper/2009.14286/) (2020)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration](https://mlsanity.com/paper/2101.10588/) (2021)
- [4+3 Phases of Compute-Optimal Neural Scaling Laws](https://mlsanity.com/paper/2405.15074/) (2024)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [Dimension free ridge regression](https://mlsanity.com/paper/2210.08571/) (2022)
- [Emergence and scaling laws in SGD learning of shallow neural networks](https://mlsanity.com/paper/2504.19983/) (2025)
- [Deterministic equivalent and error universality of deep random features learning](https://mlsanity.com/paper/2302.00401/) (2023)
- [How rotational invariance of common kernels prevents generalization in high dimensions](https://mlsanity.com/paper/2104.04244/) (2021)
- [Asymptotics of Random Feature Regression Beyond the Linear Scaling Regime](https://mlsanity.com/paper/2403.08160/) (2024)
- [Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws](https://mlsanity.com/paper/2508.03688/) (2025)
- [Dimension-free deterministic equivalents and scaling laws for random feature regression](https://mlsanity.com/paper/2405.15699/) (2024)
- [Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks](https://mlsanity.com/paper/2109.09304/) (2021)
- [Dimension-adapted Momentum Outscales SGD](https://mlsanity.com/paper/2505.16098/) (2025)
- [Covariance Estimation: Optimal Dimension-free Guarantees for Adversarial Corruption and Heavy Tails](https://mlsanity.com/paper/2205.08494/) (2022)
- [Asymptotics of Learning with Deep Structured (Random) Features](https://mlsanity.com/paper/2402.13999/) (2024)
- [Universality of Kernel Random Matrices and Kernel Regression in the Quadratic Regime](https://mlsanity.com/paper/2408.01062/) (2024)
- [Kernel ridge regression under power-law data: spectrum and generalization](https://mlsanity.com/paper/2510.04780/) (2025)
- [Universal One-third Time Scaling in Learning Peaked Distributions](https://mlsanity.com/paper/2602.03685/) (2026)
- [Accelerated SGD for Non-Strongly-Convex Least Squares](https://mlsanity.com/paper/2203.01744/) (2022)
- [Logarithmic-time Schedules for Scaling Language Models with Momentum](https://mlsanity.com/paper/2602.05298/) (2026)
- [Overparameterized random feature regression with nearly orthogonal data](https://mlsanity.com/paper/2211.06077/) (2022)
- [Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Model](https://mlsanity.com/paper/2602.04774/) (2026)

## Built on by (most cited first, 4 of 4)

- [Asymmetric Scaling Laws from Sparse Features](https://mlsanity.com/paper/2605.23591/) (2026)
- [Learning reshapes power-law anisotropy in internal representations](https://mlsanity.com/paper/2608.15239/) (2026)
- [Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropy](https://mlsanity.com/paper/2608.28564/) (2026)
- [Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulas](https://mlsanity.com/paper/2609.32695/) (2026)

## Abstract (arXiv)

Scaling laws for neural networks, in which the loss decays as a power-law in the number of parameters, data, and compute, depend fundamentally on the spectral structure of the data covariance, with power-law eigenvalue decay appearing ubiquitously in vision and language tasks. A central question is whether this spectral structure is preserved or destroyed when data passes through the basic building block of a neural network: a random linear projection followed by a nonlinear activation. We study this question for the random feature model: given data x ~ N(0,H)∈ ℝv where H has α-power-law spectrum (λj(H ) asymp j-α, α > 1), a Gaussian sketch matrix W ∈ ℝv× d, and an entrywise monomial f(y) = yp, we characterize the eigenvalues of the population random-feature covariance mathbbEx [frac1df(W^top x )otimes 2]. We prove matching upper and lower bounds: for all 1 ≤ j ≤ c1 d log-(p+1)(d), the j-th eigenvalue is of order left(logp-1(j+1)/jright)α. For  c1 d log-(p+1)(d)≤ j≤ d, the j-th eigenvalue is of order j-α up to a polylog factor. That is, the power-law exponent α is inherited exactly from the input covariance, modified only by a logarithmic correction that depends on the monomial degree p. The proof combines a dyadic head-tail decomposition with Wick chaos expansions for higher-order monomials and random matrix concentration inequalities.