mlsanityData to 7 Oct 2026

Implicit Regularization of Random Feature Models

arXiv 2002.08404 ↗First posted 19 Feb 2020stat.ML cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

Random Feature (RF) models are used as efficient parametric approximations of kernel methods. We investigate, by means of random matrix theory, the connection between Gaussian RF models and Kernel Ridge Regression (KRR). For a Gaussian RF model with P features, N data points, and a ridge λ, we show that the average (i.e. expected) RF predictor is close to a KRR predictor with an effective ridge tildeλ. We show that tildeλ > λ and tildeλ searrow λ monotonically as P grows, thus revealing the implicit regularization effect of finite RF sampling. We then compare the risk (i.e. test error) of the tildeλ-KRR predictor with the average risk of the λ-RF predictor and obtain a precise and explicit bound on their difference. Finally, we empirically find an extremely good agreement between the test errors of the average λ-RF predictor and tildeλ-KRR predictor.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 14; 2022: 14; 2023: 9; 2024: 12; 2025: 3; 2026: 1.

54 papers.

Sort:

By area:

  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  3. 2023High-dimensional analysis of double descent for linear regression with random projectionscited by 47
  4. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  5. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  6. 2021Scaling Neural Tangent Kernels via Sketching and Random Featurescited by 32
  7. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  8. 2022On the Impossible Safety of Large AI Modelscited by 27
  9. 2021Locality defeats the curse of dimensionality in convolutional teacher-student scenarioscited by 26
  10. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  11. 2021Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networkscited by 21
  12. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
38 more
  1. 2022Fast Neural Kernel Embeddings for General Activationscited by 18
  2. 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17
  3. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  4. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  5. 2024Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanismcited by 15
  6. 2023A Spectral Theory of Neural Prediction and Alignmentcited by 14
  7. 2021Exact Gap between Generalization Error and Uniform Convergence in Random Feature Modelscited by 13
  8. 2021On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Modelscited by 13
  9. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  10. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  11. 2022Generalization Through The Lens Of Leave-One-Out Errorcited by 10
  12. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 10
  13. 2021SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruningcited by 9
  14. 2022Applications of Signature Methods to Market Anomaly Detectioncited by 9
  15. 2022Regularization-wise double descent: Why it occurs and how to eliminate itcited by 9
  16. 2022Quantitative deterministic equivalent of sample covariance matrices with a general dependence structurecited by 9
  17. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 9
  18. 2023A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regressioncited by 8
  19. 2024No Free Prune: Information-Theoretic Barriers to Pruning at Initializationcited by 8
  20. 2021Random matrices in service of ML footprint: ternary random features with no performance losscited by 7
  21. 2022Bias-variance decomposition of overparameterized regression with random linear featurescited by 7
  22. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 7
  23. 2021Random Features for the Neural Tangent Kernelcited by 6
  24. 2022Overparameterized random feature regression with nearly orthogonal datacited by 6
  25. 2024Theoretical Limitations of Ensembles in the Age of Overparameterizationcited by 6
  26. 2021The Geometry of Over-parameterized Regression and Adversarial Perturbationscited by 4
  27. 2021Trap of Feature Diversity in the Learning of MLPscited by 4
  28. 2022Towards a Unified Framework for Uncertainty-aware Nonlinear Variable Selection with Theoretical Guaranteescited by 4
  29. 2022SPADE4: Sparsity and Delay Embedding based Forecasting of Epidemicscited by 4
  30. 2023On the Size and Approximation Error of Distilled Setscited by 3
  31. 2024"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approachcited by 3
  32. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  33. 2023How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layercited by 2
  34. 2023Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimalitycited by 2
  35. 2024Generalization for Least Squares Regression With Simple Spiked Covariancescited by 2
  36. 2025Spectral Analysis of Representational Similarity with Limited Neuronscited by 2
  37. 2021Asymptotic Risk of Overparameterized Likelihood Models: Double Descent Theory for Deep Neural Networkscited by 1
  38. 2023Orthogonal Random Features: Explicit Forms and Sharp Inequalitiescited by 1

50 most cited of 54 All 54 (JSON)