mlsanityData to 7 Oct 2026

Ridge interpolators in correlated factor regression models -- exact risk analysis

arXiv 2406.09183 ↗First posted 13 Jun 2024stat.ML cs.IT cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider correlated factor regression models (FRM) and analyze the performance of classical ridge interpolators. Utilizing powerful Random Duality Theory (RDT) mathematical engine, we obtain precise closed form characterizations of the underlying optimization problems and all associated optimizing quantities. In particular, we provide excess prediction risk characterizations that clearly show the dependence on all key model parameters, covariance matrices, loadings, and dimensions. As a function of the over-parametrization ratio, the generalized least squares (GLS) risk also exhibits the well known double-descent (non-monotonic) behavior. Similarly to the classical linear regression models (LRM), we demonstrate that such FRM phenomenon can be smoothened out by the optimally tuned ridge regularization. The theoretical results are supplemented by numerical simulations and an excellent agrement between the two is observed. Moreover, we note that ``ridge smootenhing'' is often of limited effect already for over-parametrization ratios above 5 and of virtually no effect for those above 10. This solidifies the notion that one of the recently most popular neural networks paradigms -- zero-training (interpolating) generalizes well -- enjoys wider applicability, including the one within the FRM estimation/prediction context.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 75 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

34 of 75 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  9. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  10. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  11. 2017Implicit Regularization in Matrix Factorizationcited by 318
  12. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
22 more
  1. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  2. 2019Two models of double descent for weak featurescited by 194
  3. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  4. 2018To understand deep learning we need to understand kernel learningcited by 186
  5. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  6. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  7. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  8. 2019Harmless interpolation of noisy data in regressioncited by 121
  9. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  10. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  11. 2018Does data interpolation contradict statistical optimality?cited by 95
  12. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  13. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  14. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  15. 2013Regularly random dualitycited by 23
  16. 2013Another look at the Gardner problemcited by 23
  17. 2013Discrete perceptronscited by 14
  18. 2019Consistent Risk Estimation in Moderately High-Dimensional Linear Regressioncited by 14
  19. 2013Upper-bounding ℓ1-optimization weak thresholdscited by 13
  20. 2013Meshes that trap random subspacescited by 8
  21. 2013A performance analysis framework for SOCP algorithms in noisy compressed sensingcited by 4
  22. 2013A problem dependent analysis of SOCP algorithms in noisy compressed sensingcited by 2

Built on by

Not cited yet.