mlsanityData to 14 Oct 2026

Power-Law Spectrum of the Random Feature Model

arXiv 2603.14578 ↗First posted 15 Mar 2026stat.ML cs.LG math.PRcited by 4

Not on the map. Why?

Abstract (arXiv)

Scaling laws for neural networks, in which the loss decays as a power-law in the number of parameters, data, and compute, depend fundamentally on the spectral structure of the data covariance, with power-law eigenvalue decay appearing ubiquitously in vision and language tasks. A central question is whether this spectral structure is preserved or destroyed when data passes through the basic building block of a neural network: a random linear projection followed by a nonlinear activation. We study this question for the random feature model: given data x ~ N(0,H)∈ ℝv where H has α-power-law spectrum (λj(H ) asymp j-α, α > 1), a Gaussian sketch matrix W ∈ ℝv× d, and an entrywise monomial f(y) = yp, we characterize the eigenvalues of the population random-feature covariance mathbbEx [frac1df(W^top x )otimes 2]. We prove matching upper and lower bounds: for all 1 ≤ j ≤ c1 d log-(p+1)(d), the j-th eigenvalue is of order left(logp-1(j+1)/jright)α. For c1 d log-(p+1)(d)≤ j≤ d, the j-th eigenvalue is of order j-α up to a polylog factor. That is, the power-law exponent α is inherited exactly from the input covariance, modified only by a logarithmic correction that depends on the monomial degree p. The proof combines a dyadic head-tail decomposition with Wick chaos expansions for higher-order monomials and random matrix concentration inequalities.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 32 of its 59 references that are in the map. First posted on arXiv 15 Mar 2026.

Builds on

32 of 59 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2019Benign Overfitting in Linear Regressioncited by 549
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  6. 2021Explaining Neural Scaling Lawscited by 363
  7. 2021Deep learning: a statistical viewpointcited by 240
  8. 2020Benign overfitting in ridge regressioncited by 153
  9. 2022A Solvable Model of Neural Scaling Lawscited by 116
  10. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  11. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 68
  12. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 57
20 more
  1. 2022Dimension free ridge regressioncited by 51
  2. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  3. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  4. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 32
  5. 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 26
  6. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 25
  7. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  8. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  9. 2022Covariance Estimation: Optimal Dimension-free Guarantees for Adversarial Corruption and Heavy Tailscited by 14
  10. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  11. 2024Universality of Kernel Random Matrices and Kernel Regression in the Quadratic Regimecited by 12
  12. 2025Dimension-adapted Momentum Outscales SGDcited by 12
  13. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 10
  14. 2026Universal One-third Time Scaling in Learning Peaked Distributionscited by 10
  15. 2022Accelerated SGD for Non-Strongly-Convex Least Squarescited by 9
  16. 2026Logarithmic-time Schedules for Scaling Language Models with Momentumcited by 9
  17. 2022Overparameterized random feature regression with nearly orthogonal datacited by 7
  18. 2026Theory of Optimal Learning Rate Schedules and Scaling Laws for a Random Feature Modelcited by 7
  19. 2025A general technique for approximating high-dimensional empirical kernel matricescited by 4
  20. 2025On the Effectiveness of the z-Transform Method in Quadratic Optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Asymmetric Scaling Laws from Sparse Featurescited by 1
  2. 2026Learning reshapes power-law anisotropy in internal representationscited by 0
  3. 2026Learning between the peaks: sharp asymptotics for kernel ridge regression under power-law anisotropycited by 0
  4. 2026Affine Geometry of Gaussian ReLU Networks via Conditional Kac-Rice Formulascited by 0