mlsanityData to 7 Oct 2026

Kernel ridge regression under power-law data: spectrum and generalization

arXiv 2510.04780 ↗First posted 6 Oct 2025stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

In this work, we investigate high-dimensional kernel ridge regression (KRR) on i.i.d. Gaussian data with anisotropic power-law covariance. This setting differs fundamentally from the classical source & capacity conditions for KRR, where power-law assumptions are typically imposed on the kernel eigen-spectrum itself. Our contributions are twofold. First, we derive an explicit characterization of the kernel spectrum for polynomial inner-product kernels, giving a precise description of how the kernel eigen-spectrum inherits the data decay. Second, we provide an asymptotic analysis of the excess risk in the high-dimensional regime for a particular kernel with this spectral behavior, showing that the sample complexity is governed by the effective dimension of the data rather than the ambient dimension. These results establish a fundamental advantage of learning with power-law anisotropic data over isotropic data. To our knowledge, this is the first rigorous treatment of non-linear KRR under power-law data.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

37 of 63 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2018On Lazy Training in Differentiable Programmingcited by 671
  5. 2017Deep Neural Networks as Gaussian Processescited by 617
  6. 2019Benign Overfitting in Linear Regressioncited by 544
  7. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  8. 2021Explaining Neural Scaling Lawscited by 353
  9. 2016Generalization Properties of Learning with Random Featurescited by 163
  10. 2019Linearized two-layers neural networks in high dimensioncited by 155
  11. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  12. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
25 more
  1. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  2. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  3. 2022A Solvable Model of Neural Scaling Lawscited by 101
  4. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  5. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  6. 2018Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passescited by 69
  7. 20244+3 Phases of Compute-Optimal Neural Scaling Lawscited by 66
  8. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  9. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  10. 2022Dimension free ridge regressioncited by 50
  11. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  12. 2018Learning with SGD and Random Featurescited by 33
  13. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  14. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  15. 2025Emergence and scaling laws in SGD learning of shallow neural networkscited by 29
  16. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  17. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  18. 2024Dimension-free deterministic equivalents and scaling laws for random feature regressioncited by 23
  19. 2025Learning quadratic neural networks in high dimensions: SGD dynamics and scaling lawscited by 23
  20. 2025Scaling Laws for Gradient Descent and Sign Descent for Linear Bigram Models under Zipf's Lawcited by 19
  21. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  22. 2022Error Scaling Laws for Kernel Classification under Source and Capacity Conditionscited by 17
  23. 2022Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional datacited by 12
  24. 2024Universality of Kernel Random Matrices and Kernel Regression in the Quadratic Regimecited by 11
  25. 2023Nonparametric Linear Feature Learning in Regression Through Regularisationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026Optimal scaling laws in learning hierarchical multi-index modelscited by 6
  2. 2026Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase Retrievalcited by 5
  3. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  4. 2026Power-Law Spectrum of the Random Feature Modelcited by 3
  5. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0
  6. 2026Asymmetric Scaling Laws from Sparse Featurescited by 0
  7. 2026Learning reshapes power-law anisotropy in internal representationscited by 0
  8. 2026First Learn, Then Memorize: The Spectral Bias of Diffusion Modelscited by 0