# Estimation of prediction error with known covariate shift

arXiv: https://arxiv.org/abs/2205.01849 · HTML: https://mlsanity.com/paper/2205.01849/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2022.

## Built on by (most cited first, 2 of 2)

- [Large Language Model Routing with Benchmark Datasets](https://mlsanity.com/paper/2309.15789/) (2023)
- [Understanding new tasks through the lens of training data via exponential tilting](https://mlsanity.com/paper/2205.13577/) (2022)

## Abstract (arXiv)

In supervised learning, the estimation of prediction error on unlabeled test data is an important task. Existing methods are usually built on the assumption that the training and test data are sampled from the same distribution, which is often violated in practice. As a result, traditional estimators like cross-validation (CV) will be biased and this may result in poor model selection. In this paper, we assume that we have a test dataset in which the feature values are available but not the outcome labels, and focus on a particular form of distributional shift called "covariate shift". We propose an alternative method based on parametric bootstrap of the target of conditional error. Empirically, our method outperforms CV for both simulation and real data example across different modeling tasks.