# On the cross-validation bias due to unsupervised pre-processing

arXiv: https://arxiv.org/abs/1901.08974 · HTML: https://mlsanity.com/paper/1901.08974/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jan 2019.

## Built on by (most cited first, 5 of 5)

- [AIRepr: An Analyst-Inspector Framework for Evaluating Reproducibility of LLMs in Data Science](https://mlsanity.com/paper/2502.16395/) (2025)
- [Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classification](https://mlsanity.com/paper/2410.00179/) (2024)
- [What is to be gained by ensemble models in analysis of spectroscopic data?](https://mlsanity.com/paper/2404.02184/) (2024)
- [bioLeak: Leakage-Aware Modeling and Diagnostics for Machine Learning in R](https://mlsanity.com/paper/2604.10965/) (2026)
- [When a High Score Is an Illusion: Certifying Genuine versus Repackaged Forecasting Skill](https://mlsanity.com/paper/2609.19223/) (2026)

## Abstract (arXiv)

Cross-validation is the de facto standard for predictive model evaluation and selection. In proper use, it provides an unbiased estimate of a model's predictive performance. However, data sets often undergo various forms of data-dependent preprocessing, such as mean-centering, rescaling, dimensionality reduction, and outlier removal. It is often believed that such preprocessing stages, if done in an unsupervised manner (that does not incorporate the class labels or response values) are generally safe to do prior to cross-validation. In this paper, we study three commonly-practiced preprocessing procedures prior to a regression analysis: (i) variance-based feature selection; (ii) grouping of rare categorical features; and (iii) feature rescaling. We demonstrate that unsupervised preprocessing can, in fact, introduce a substantial bias into cross-validation estimates and potentially hurt model selection. This bias may be either positive or negative and its exact magnitude depends on all the parameters of the problem in an intricate manner. Further research is needed to understand the real-world impact of this bias across different application domains, particularly when dealing with small sample sizes and high-dimensional data.