# Covariate Shift in High-Dimensional Random Feature Regression

arXiv: https://arxiv.org/abs/2111.08234 · HTML: https://mlsanity.com/paper/2111.08234/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2021.

## Built on by (most cited first, 24 of 24)

- [High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation](https://mlsanity.com/paper/2205.01445/) (2022)
- [Strong Model Collapse](https://mlsanity.com/paper/2410.04840/) (2024)
- [Predicting Out-of-Distribution Error with the Projection Norm](https://mlsanity.com/paper/2202.05834/) (2022)
- [Uncertainty-Informed Deep Learning Models Enable High-Confidence Predictions for Digital Histopathology](https://mlsanity.com/paper/2204.04516/) (2022)
- [Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties](https://mlsanity.com/paper/2205.07069/) (2022)
- [Nonlinear spiked covariance matrices and signal propagation in deep neural networks](https://mlsanity.com/paper/2402.10127/) (2024)
- [Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions](https://mlsanity.com/paper/2206.07252/) (2022)
- [Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression](https://mlsanity.com/paper/2205.14846/) (2022)
- [Optimal Ridge Regularization for Out-of-Distribution Prediction](https://mlsanity.com/paper/2404.01233/) (2024)
- [Investigating Power laws in Deep Representation Learning](https://mlsanity.com/paper/2202.05808/) (2022)
- [Pseudo-Labeling for Kernel Ridge Regression under Covariate Shift](https://mlsanity.com/paper/2302.10160/) (2023)
- [Loss-to-Loss Prediction: Scaling Laws for All Datasets](https://mlsanity.com/paper/2411.12925/) (2024)
- [On Regularization via Early Stopping for Least Squares Regression](https://mlsanity.com/paper/2406.04425/) (2024)
- [Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks](https://mlsanity.com/paper/2310.20579/) (2023)
- [Accuracy on the wrong line: On the pitfalls of noisy data for out-of-distribution generalisation](https://mlsanity.com/paper/2406.19049/) (2024)
- [Risk and cross validation in ridge regression with correlated samples](https://mlsanity.com/paper/2408.04607/) (2024)
- [Pretrain-Test Task Alignment Governs Generalization in In-Context Learning](https://mlsanity.com/paper/2509.26551/) (2025)
- [Least Squares Regression Can Exhibit Under-Parameterized Double Descent](https://mlsanity.com/paper/2305.14689/) (2023)
- [High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization](https://mlsanity.com/paper/2406.03171/) (2024)
- [Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure](https://mlsanity.com/paper/2503.00856/) (2025)
- [Sharp analysis of out-of-distribution error for "importance-weighted" estimators in the overparameterized regime](https://mlsanity.com/paper/2405.06546/) (2024)
- [Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization](https://mlsanity.com/paper/2210.11589/) (2022)
- [Transfer learning via Regularized Linear Discriminant Analysis](https://mlsanity.com/paper/2501.02411/) (2025)
- [Prediction Accuracy & Reliability: Classification and Object Localization under Distribution Shift](https://mlsanity.com/paper/2409.03543/) (2024)

## Abstract (arXiv)

A significant obstacle in the development of robust machine learning models is covariate shift, a form of distribution shift that occurs when the input distributions of the training and test sets differ while the conditional label distributions remain the same. Despite the prevalence of covariate shift in real-world applications, a theoretical understanding in the context of modern machine learning has remained lacking. In this work, we examine the exact high-dimensional asymptotics of random feature regression under covariate shift and present a precise characterization of the limiting test error, bias, and variance in this setting. Our results motivate a natural partial order over covariate shifts that provides a sufficient condition for determining when the shift will harm (or even help) test performance. We find that overparameterized models exhibit enhanced robustness to covariate shift, providing one of the first theoretical explanations for this intriguing phenomenon. Additionally, our analysis reveals an exact linear relationship between in-distribution and out-of-distribution generalization performance, offering an explanation for this surprising recent empirical observation.