# Revisiting Optimism and Model Complexity in the Wake of Overparameterized Machine Learning

arXiv: https://arxiv.org/abs/2410.01259 · HTML: https://mlsanity.com/paper/2410.01259/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 91 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 15 of 15)

- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Dimension free ridge regression](https://mlsanity.com/paper/2210.08571/) (2022)
- [The Lasso with general Gaussian designs with applications to hypothesis testing](https://mlsanity.com/paper/2007.13716/) (2020)
- [Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting](https://mlsanity.com/paper/2207.06569/) (2022)
- [A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning](https://mlsanity.com/paper/2310.18988/) (2023)
- [Optimal Ridge Regularization for Out-of-Distribution Prediction](https://mlsanity.com/paper/2404.01233/) (2024)
- [Subsample Ridge Ensembles: Equivalences and Generalized Cross-Validation](https://mlsanity.com/paper/2304.13016/) (2023)
- [Mitigating multiple descents: A model-agnostic framework for risk monotonization](https://mlsanity.com/paper/2205.12937/) (2022)
- [Generalized equivalences between subsampling and ridge regularization](https://mlsanity.com/paper/2305.18496/) (2023)
- [Minimum ℓ1-norm interpolators: Precise asymptotics and multiple descent](https://mlsanity.com/paper/2110.09502/) (2021)
- [Unbiased Risk Estimation in the Normal Means Problem via Coupled Bootstrap Techniques](https://mlsanity.com/paper/2111.09447/) (2021)
- [An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression](https://mlsanity.com/paper/2306.13185/) (2023)
- [Precise Asymptotics of Bagging Regularized M-estimators](https://mlsanity.com/paper/2409.15252/) (2024)
- [Unbiased Test Error Estimation in the Poisson Means Problem via Coupled Bootstrap Techniques](https://mlsanity.com/paper/2212.01943/) (2022)

## Built on by (most cited first, 2 of 2)

- [Is Supervised Learning Really That Different from Unsupervised?](https://mlsanity.com/paper/2505.11006/) (2025)
- [A Rigorous, Tractable Measure of Model Complexity](https://mlsanity.com/paper/2605.21167/) (2026)

## Abstract (arXiv)

Common practice in modern machine learning involves fitting a large number of parameters relative to the number of observations. These overparameterized models can exhibit surprising generalization behavior, e.g., ``double descent'' in the prediction error curve when plotted against the raw number of model parameters, or another simplistic notion of complexity. In this paper, we revisit model complexity from first principles, by first reinterpreting and then extending the classical statistical concept of (effective) degrees of freedom. Whereas the classical definition is connected to fixed-X prediction error (in which prediction error is defined by averaging over the same, nonrandom covariate points as those used during training), our extension of degrees of freedom is connected to random-X prediction error (in which prediction error is averaged over a new, random sample from the covariate distribution). The random-X setting more naturally embodies modern machine learning problems, where highly complex models, even those complex enough to interpolate the training data, can still lead to desirable generalization performance under appropriate conditions. We demonstrate the utility of our proposed complexity measures through a mix of conceptual arguments, theory, and experiments, and illustrate how they can be used to interpret and compare arbitrary prediction models.