mlsanityData to 7 Oct 2026

Dimension-free deterministic equivalents and scaling laws for random feature regression

arXiv 2405.15699 ↗First posted 24 May 2024stat.ML cond-mat.dis-nn cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

In this work we investigate the generalization performance of random feature ridge regression (RFRR). Our main contribution is a general deterministic equivalent for the test error of RFRR. Specifically, under a certain concentration property, we show that the test error is well approximated by a closed-form expression that only depends on the feature map eigenvalues. Notably, our approximation guarantee is non-asymptotic, multiplicative, and independent of the feature map dimension -- allowing for infinite-dimensional features. We expect this deterministic equivalent to hold broadly beyond our theoretical analysis, and we empirically validate its predictions on various real and synthetic datasets. As an application, we derive sharp excess error rates under standard power-law assumptions of the spectrum and target decay. In particular, we provide a tight result for the smallest number of features achieving optimal minimax error rate.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 34 of its 65 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

34 of 65 references.

References · 34

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  7. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  8. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  9. 2024A Dynamical Model of Neural Scaling Lawscited by 105
  10. 2022A Solvable Model of Neural Scaling Lawscited by 101
  11. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  12. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
22 more
  1. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  2. 2020Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networkscited by 76
  3. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  4. 2022Dimension free ridge regressioncited by 50
  5. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 46
  6. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  7. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  8. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 35
  9. 2018On the Spectrum of Random Features Maps of High Dimensional Datacited by 27
  10. 2021The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networkscited by 25
  11. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 25
  12. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  13. 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 22
  14. 2024A non-asymptotic theory of Kernel Ridge Regression: deterministic equivalents, test error, and GCV estimatorcited by 22
  15. 2021Learning with convolution and pooling operations in kernel methodscited by 21
  16. 2024An exactly solvable model for emergence and scaling laws in the multitask sparse parity problemcited by 21
  17. 2023On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decaycited by 19
  18. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  19. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  20. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  21. 2022Quantitative deterministic equivalent of sample covariance matrices with a general dependence structurecited by 9
  22. 2022Random Features Model with General Convex Regularization: A Fine Grained Analysis with Precise Asymptotic Learning Curvescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 8.

23 papers.

Sort:

  1. 2024Scaling Laws in Linear Regression: Compute, Parameters, and Datacited by 54
  2. 2025Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regimecited by 18
  3. 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
  4. 2025Weak-to-Strong Generalization Even in Random Feature Networks, Provablycited by 13
  5. 2025Models of Heavy-Tailed Mechanistic Universalitycited by 9
  6. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 8
  7. 2026Optimal scaling laws in learning hierarchical multi-index modelscited by 6
  8. 2025Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Lawscited by 5
  9. 2025A Law of Data Reconstruction for Random Features (and Beyond)cited by 3
  10. 2025Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Datacited by 3
  11. 2026Power-Law Spectrum of the Random Feature Modelcited by 3
  12. 2025Physics of Learning: A Lagrangian perspective to different learning paradigmscited by 2
11 more
  1. 2026Deep Learning of Compositional Targets with Hierarchical Spectral Methodscited by 2
  2. 2024No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditionscited by 1
  3. 2025High-dimensional ridge regression with random features for non-identically distributed data with a variance profilecited by 1
  4. 2025On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classificationcited by 1
  5. 2025Neural Scaling Laws for Deep Regressioncited by 1
  6. 2025Breaking the curse of dimensionality for linear rules: optimal predictors over the ellipsoidcited by 1
  7. 2026A Random Matrix Theory of Masked Self-Supervised Regressioncited by 0
  8. 2026Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networkscited by 0
  9. 2026Asymmetric Scaling Laws from Sparse Featurescited by 0
  10. 2026How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscited by 0
  11. 2026Neural scaling laws and evolution of learnable activation functions of Kolmogorov-Arnold networkscited by 0