# The out-of-sample R2: estimation and inference

arXiv: https://arxiv.org/abs/2302.05131 · HTML: https://mlsanity.com/paper/2302.05131/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 30 references that are in the map.
- First posted on arXiv 10 Feb 2023.

## Builds on (most cited first, 1 of 1)

- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)

## Built on by (most cited first, 1 of 1)

- [What Are Large Language Models Mapping to in the Brain? A Case Against Over-Reliance on Brain Scores](https://mlsanity.com/paper/2406.01538/) (2024)

## Abstract (arXiv)

Out-of-sample prediction is the acid test of predictive models, yet an independent test dataset is often not available for assessment of the prediction error. For this reason, out-of-sample performance is commonly estimated using data splitting algorithms such as cross-validation or the bootstrap. For quantitative outcomes, the ratio of variance explained to total variance can be summarized by the coefficient of determination or in-sample R2, which is easy to interpret and to compare across different outcome variables. As opposed to the in-sample R2, the out-of-sample R2 has not been well defined and the variability on the out-of-sample hatR2 has been largely ignored. Usually only its point estimate is reported, hampering formal comparison of predictability of different outcome variables. Here we explicitly define the out-of-sample R2 as a comparison of two predictive models, provide an unbiased estimator and exploit recent theoretical advances on uncertainty of data splitting estimates to provide a standard error for the hatR2. The performance of the estimators for the R2 and its standard error are investigated in a simulation study. We demonstrate our new method by constructing confidence intervals and comparing models for prediction of quantitative Brassica napus and Zea mays phenotypes based on gene expression data.