mlsanityData to 7 Oct 2026

Random features and polynomial rules

arXiv 2402.10164 ↗First posted 15 Feb 2024cond-mat.dis-nn cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Random features models play a distinguished role in the theory of deep learning, describing the behavior of neural networks close to their infinite-width limit. In this work, we present a thorough analysis of the generalization performance of random features models for generic supervised learning problems with Gaussian data. Our approach, built with tools from the statistical mechanics of disordered systems, maps the random features model to an equivalent polynomial model, and allows us to plot average generalization curves as functions of the two main control parameters of the problem: the number of random features N and the size P of the training set, both assumed to scale as powers in the input dimension D. Our results extend the case of proportional scaling between N, P and D. They are in accordance with rigorous bounds known for certain particular learning tasks and are in quantitative agreement with numerical experiments performed over many order of magnitudes of N and P. We find good agreement also far from the asymptotic limits where D→ ∞ and at least one between P/DK, N/DL remains finite.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 40 of its 92 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

40 of 92 references.

References · 40

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2017Deep Neural Networks as Gaussian Processescited by 617
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
  7. 2021Deep learning: a statistical viewpointcited by 239
  8. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  9. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  10. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  11. 2019Linearized two-layers neural networks in high dimensioncited by 155
  12. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
28 more
  1. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  2. 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
  3. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  4. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  5. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  6. 2021Learning with invariances in random features and kernel modelscited by 89
  7. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  8. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  9. 2022Universality of empirical risk minimizationcited by 73
  10. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 70
  11. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 67
  12. 2020Triple descent and the two kinds of overfitting: Where & why do they appear?cited by 64
  13. 2017Classification and Geometry of General Perceptual Manifoldscited by 62
  14. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  15. 2022Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regressioncited by 42
  16. 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
  17. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  18. 2021A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNscited by 33
  19. 2022An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalingscited by 27
  20. 2022The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimescited by 25
  21. 2022Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regressioncited by 22
  22. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  23. 2022Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regimecited by 18
  24. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  25. 2015Linear Readout of Object Manifoldscited by 9
  26. 2020Beyond the storage capacity: data driven satisfiability transitioncited by 6
  27. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  28. 2020Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networkscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

  1. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  2. 2024Feature learning in finite-width Bayesian deep linear networks with multiple outputs and convolutional layerscited by 12
  3. 2024High-dimensional learning of narrow neural networkscited by 12
  4. 2025Statistical mechanics of extensive-width Bayesian neural networks near interpolationcited by 9
  5. 2025Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolationcited by 0
  6. 2025Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolationcited by 0