mlsanityData to 7 Oct 2026

High-dimensional analysis of double descent for linear regression with random projections

arXiv 2303.01372 ↗First posted 2 Mar 2023cs.LG stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

We consider linear regression problems with a varying number of random projections, where we provably exhibit a double descent curve for a fixed prediction problem, with a high-dimensional analysis based on random matrix theory. We first consider the ridge regression estimator and review earlier results using classical notions from non-parametric statistics, namely degrees of freedom, also known as effective dimensionality. We then compute asymptotic equivalents of the generalization performance (in terms of squared bias and variance) of the minimum norm least-squares fit with random projections, providing simple expressions for the double descent phenomenon.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

24 of 44 references.

References · 24

  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  2. 2019Benign Overfitting in Linear Regressioncited by 544
  3. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  6. 2021Deep learning: a statistical viewpointcited by 239
  7. 2019Two models of double descent for weak featurescited by 194
  8. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  9. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  10. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  11. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  12. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
12 more
  1. 2020A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descentcited by 69
  2. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  3. 2020Implicit Regularization of Random Feature Modelscited by 54
  4. 2022Dimension free ridge regressioncited by 50
  5. 2020A Brief Prehistory of Double Descentcited by 44
  6. 2021Hessian Eigenspectra of More Realistic Nonlinear Modelscited by 33
  7. 2014A Statistical Perspective on Randomized Sketching for Ordinary Least-Squarescited by 21
  8. 2022Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimensioncited by 19
  9. 2022An elementary analysis of ridge regression with random designcited by 15
  10. 2022Asymptotics of the Sketched Pseudoinversecited by 9
  11. 2023Sketched Ridgeless Linear Regression: The Role of Downsamplingcited by 7
  12. 2021The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regressioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 19; 2025: 16; 2026: 4.

47 papers.

Sort:

By area:

  1. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  2. 2024Strong Model Collapsecited by 62
  3. 2024Scaling and renormalization in high-dimensional regressioncited by 45
  4. 2023Generalization in Kernel Regression Under Realistic Assumptionscited by 26
  5. 2023Regularization properties of adversarially-trained linear regressioncited by 17
  6. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  7. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  8. 2024Asymptotics of Learning with Deep Structured (Random) Featurescited by 13
  9. 2025Rethinking Early Stopping: Refine, Then Calibratecited by 13
  10. 2023Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuningcited by 11
  11. 2025Asymptotics of Non-Convex Generalized Linear Models in High-Dimensions: A proof of the replica formulacited by 11
  12. 2023A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regressioncited by 8
35 more
  1. 2024Towards an Improved Understanding and Utilization of Maximum Manifold Capacity Representationscited by 8
  2. 2024Risk and cross validation in ridge regression with correlated samplescited by 8
  3. 2024LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learningcited by 8
  4. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  5. 2025Preventing Model Collapse Under Overparametrization: Optimal Mixing Ratios for Interpolation Learning and Ridge Regressioncited by 7
  6. 2024Theoretical Limitations of Ensembles in the Age of Overparameterizationcited by 6
  7. 2024A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regressioncited by 6
  8. 2025Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Modelscited by 6
  9. 2023An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regressioncited by 5
  10. 2024High-dimensional analysis of ridge regression for non-identically distributed data with a variance profilecited by 4
  11. 2024Safety vs. Performance: How Multi-Objective Learning Reduces Barriers to Market Entrycited by 4
  12. 2023Does Double Descent Occur in Self-Supervised Learning?cited by 3
  13. 2024Generative Bayesian Computation for Maximum Expected Utilitycited by 3
  14. 2025Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarkscited by 3
  15. 2025Learning from one graph: transductive learning guarantees via the geometry of small random worldscited by 3
  16. 2024A solvable generative model with a linear, one-step denoisercited by 2
  17. 2024Generative Modeling: A Reviewcited by 2
  18. 2025On the Effectiveness of the z-Transform Method in Quadratic Optimizationcited by 2
  19. 2025Understanding Overparametrization in Survival Models through Interpolationcited by 2
  20. 2023How many samples are needed to leverage smoothness?cited by 1
  21. 2024Implicit Regularization Paths of Weighted Neural Representationscited by 1
  22. 2024No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditionscited by 1
  23. 2025Bayesian Double Descentcited by 1
  24. 2025Breaking the curse of dimensionality for linear rules: optimal predictors over the ellipsoidcited by 1
  25. 2024Double Descent Meets Out-of-Distribution Detection: Theoretical Insights and Empirical Analysis on the role of model complexitycited by 0
  26. 2025auto-fpt: Automating Free Probability Theory Calculations for Machine Learning Theorycited by 0
  27. 2025Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoderscited by 0
  28. 2025A Random Matrix Perspective of Echo State Networks: From Precise Bias--Variance Characterization to Optimal Regularizationcited by 0
  29. 2025Calibrated Principal Component Regressioncited by 0
  30. 2025Generalization in Representation Models via Random Matrix Theory: Application to Recurrent Networkscited by 0
  31. 2025Task Matrices: Linear Maps for Cross-Model Finetuning Transfercited by 0
  32. 2026Deep Exploration of Epoch-wise Double Descent in Noisy Data: Signal Separation, Large Activation, and Benign Overfittingcited by 0
  33. 2026A Random Matrix Theory of Masked Self-Supervised Regressioncited by 0
  34. 2026A Rigorous, Tractable Measure of Model Complexitycited by 0
  35. 2026Benign Overfitting for General Norms and Distributionscited by 0