mlsanityData to 7 Oct 2026

A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors

arXiv 2207.11621 ↗First posted 23 Jul 2022stat.ML cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

In this work we establish an algorithm and distribution independent non-asymptotic trade-off between the model size, excess test loss, and training loss of linear predictors. Specifically, we show that models that perform well on the test data (have low excess loss) are either "classical" -- have training loss close to the noise level, or are "modern" -- have a much larger number of parameters compared to the minimum needed to fit the training data exactly. We also provide a more precise asymptotic analysis when the limiting spectral distribution of the whitened features is Marchenko-Pastur. Remarkably, while the Marchenko-Pastur analysis is far more precise near the interpolation peak, where the number of parameters is just enough to fit the training data, it coincides exactly with the distribution independent bound as the level of overparametrization increases.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 23 Jul 2022.

Builds on

26 of 32 references.

References · 26

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  3. 2019Benign Overfitting in Linear Regressioncited by 544
  4. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
  5. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  6. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  7. 2019Two models of double descent for weak featurescited by 194
  8. 2018Just Interpolate: Kernel "Ridgeless" Regression Can Generalizecited by 190
  9. 2020Benign overfitting in ridge regressioncited by 152
  10. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  11. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 146
  12. 2019Harmless interpolation of noisy data in regressioncited by 121
14 more
  1. 2021Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentrationcited by 110
  2. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  3. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  4. 2018Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolatecited by 99
  5. 2018Does data interpolation contradict statistical optimality?cited by 95
  6. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  7. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  8. 2021Benign Overfitting of Constant-Stepsize SGD for Linear Regressioncited by 71
  9. 2021Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds, and Benign Overfittingcited by 54
  10. 2021Towards an Understanding of Benign Overfitting in Neural Networkscited by 29
  11. 2021Foolish Crowds Support Benign Overfittingcited by 24
  12. 2022Memorize to Generalize: on the Necessity of Interpolation in High Dimensional Linear Regressioncited by 14
  13. 2020On the Universality of the Double Descent Peak in Ridgeless Regressioncited by 11
  14. 2022Benign overfitting and adaptive nonparametric regressioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 2; 2025: 1; 2026: 1.

7 papers.

  1. 2023Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimensioncited by 26
  2. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 17
  3. 2023Noisy Interpolation Learning with Shallow Univariate ReLU Networkscited by 11
  4. 2024Near-Interpolators: Rapid Norm Growth and the Trade-Off between Interpolation and Generalizationcited by 7
  5. 2024Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionalitycited by 7
  6. 2025Feature maps for the Laplacian kernel and its generalizationscited by 1
  7. 2026The Price of Robustness: Stable Classifiers Need Overparameterizationcited by 0