mlsanityData to 7 Oct 2026

Precise analysis of ridge interpolators under heavy correlations -- a Random Duality Theory view

arXiv 2406.09199 ↗First posted 13 Jun 2024stat.ML cs.IT cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We consider fully row/column-correlated linear regression models and study several classical estimators (including minimum norm interpolators (GLS), ordinary least squares (LS), and ridge regressors). We show that Random Duality Theory (RDT) can be utilized to obtain precise closed form characterizations of all estimators related optimizing quantities of interest, including the prediction risk (testing or generalization error). On a qualitative level out results recover the risk's well known non-monotonic (so-called double-descent) behavior as the number of features/sample size ratio increases. On a quantitative level, our closed form results show how the risk explicitly depends on all key model parameters, including the problem dimensions and covariance matrices. Moreover, a special case of our results, obtained when intra-sample (or time-series) correlations are not present, precisely match the corresponding ones obtained via spectral methods in [6,16,17,24].

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 60 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

34 of 60 references.

References · 34

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  4. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  6. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  7. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  8. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  9. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  10. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  11. 2017Implicit Regularization in Matrix Factorizationcited by 318
  12. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
22 more
  1. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  2. 2019Two models of double descent for weak featurescited by 194
  3. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  4. 2018To understand deep learning we need to understand kernel learningcited by 186
  5. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  6. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  7. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  8. 2019Harmless interpolation of noisy data in regressioncited by 121
  9. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  10. 2020On the Optimal Weighted ℓ2 Regularization in Overparameterized Linear Regressioncited by 97
  11. 2018Does data interpolation contradict statistical optimality?cited by 95
  12. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 67
  13. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  14. 2020Asymptotics of Ridge (less) Regression under General Source Conditioncited by 61
  15. 2013Regularly random dualitycited by 23
  16. 2013Another look at the Gardner problemcited by 23
  17. 2013Discrete perceptronscited by 14
  18. 2019Consistent Risk Estimation in Moderately High-Dimensional Linear Regressioncited by 14
  19. 2013Upper-bounding ℓ1-optimization weak thresholdscited by 13
  20. 2013Meshes that trap random subspacescited by 8
  21. 2013A performance analysis framework for SOCP algorithms in noisy compressed sensingcited by 4
  22. 2013A problem dependent analysis of SOCP algorithms in noisy compressed sensingcited by 2

Built on by

Not cited yet.