# From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation

arXiv: https://arxiv.org/abs/1704.08160 · HTML: https://mlsanity.com/paper/1704.08160/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2017.

## Built on by (most cited first, 12 of 12)

- [More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize](https://mlsanity.com/paper/2203.06176/) (2022)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers](https://mlsanity.com/paper/2402.01502/) (2024)
- [Active operator inference for learning low-dimensional dynamical-system models from noisy data](https://mlsanity.com/paper/2107.09256/) (2021)
- [Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs](https://mlsanity.com/paper/2409.18842/) (2024)
- [The Limits of Assumption-free Tests for Algorithm Performance](https://mlsanity.com/paper/2402.07388/) (2024)
- [GRASP: A Goodness-of-Fit Test for Classification Learning](https://mlsanity.com/paper/2209.02064/) (2022)
- [Inadmissibility of the corrected Akaike information criterion](https://mlsanity.com/paper/2211.09326/) (2022)
- [Predictive Performance Test based on the Exhaustive Nested Cross-Validation for High-dimensional data](https://mlsanity.com/paper/2408.03138/) (2024)
- [Is Supervised Learning Really That Different from Unsupervised?](https://mlsanity.com/paper/2505.11006/) (2025)
- [A Rigorous, Tractable Measure of Model Complexity](https://mlsanity.com/paper/2605.21167/) (2026)
- [Optimal ridge regularization revisited](https://mlsanity.com/paper/2605.28679/) (2026)

## Abstract (arXiv)

In statistical prediction, classical approaches for model selection and model evaluation based on covariance penalties are still widely used. Most of the literature on this topic is based on what we call the "Fixed-X" assumption, where covariate values are assumed to be nonrandom. By contrast, it is often more reasonable to take a "Random-X" view, where the covariate values are independently drawn for both training and prediction. To study the applicability of covariance penalties in this setting, we propose a decomposition of Random-X prediction error in which the randomness in the covariates contributes to both the bias and variance components. This decomposition is general, but we concentrate on the fundamental case of least squares regression. We prove that in this setting the move from Fixed-X to Random-X prediction results in an increase in both bias and variance. When the covariates are normally distributed and the linear model is unbiased, all terms in this decomposition are explicitly computable, which yields an extension of Mallows' Cp that we call RCp. RCp also holds asymptotically for certain classes of nonnormal covariates. When the noise variance is unknown, plugging in the usual unbiased estimate leads to an approach that we call hatRCp, which is closely related to Sp (Tukey 1967), and GCV (Craven and Wahba 1978). For excess bias, we propose an estimate based on the "shortcut-formula" for ordinary cross-validation (OCV), resulting in an approach we call RCp^+. Theoretical arguments and numerical simulations suggest that RCP^+ is typically superior to OCV, though the difference is small. We further examine the Random-X error of other popular estimators. The surprising result we get for ridge regression is that, in the heavily-regularized regime, Random-X variance is smaller than Fixed-X variance, which can lead to smaller overall Random-X error.