# Extrapolated cross-validation for randomized ensembles

arXiv: https://arxiv.org/abs/2302.13511 · HTML: https://mlsanity.com/paper/2302.13511/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 65 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Mean estimation and regression under heavy-tailed distributions--a survey](https://mlsanity.com/paper/1906.04280/) (2019)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [Cross-validation: what does it estimate and how well does it do it?](https://mlsanity.com/paper/2104.00673/) (2021)
- [Multiple Descent: Design Your Own Generalization Curve](https://mlsanity.com/paper/2008.01036/) (2020)
- [Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife](https://mlsanity.com/paper/1311.4555/) (2013)
- [Cross-validation Confidence Intervals for Test Error](https://mlsanity.com/paper/2007.12671/) (2020)
- [The Implicit Regularization of Ordinary Least Squares Ensembles](https://mlsanity.com/paper/1910.04743/) (2019)
- [Asymptotic Distributions and Rates of Convergence for Random Forests via Generalized U-statistics](https://mlsanity.com/paper/1905.10651/) (2019)
- [Approximate Leave-One-Out for Fast Parameter Tuning in High Dimensions](https://mlsanity.com/paper/1807.02694/) (2018)
- [Cross-Validation with Confidence](https://mlsanity.com/paper/1703.07904/) (2017)
- [Approximate Cross-validation: Guarantees for Model Assessment and Selection](https://mlsanity.com/paper/2003.00617/) (2020)
- [Error bounds in estimating the out-of-sample prediction error using leave-one-out cross validation in high-dimensions](https://mlsanity.com/paper/2003.01770/) (2020)
- [Bagging in overparameterized learning: Risk characterization and risk monotonization](https://mlsanity.com/paper/2210.11445/) (2022)
- [Stability revisited: new generalisation bounds for the Leave-one-Out](https://mlsanity.com/paper/1608.06412/) (2016)
- [Estimating the Algorithmic Variance of Randomized Ensembles via the Bootstrap](https://mlsanity.com/paper/1907.08742/) (2019)
- [Mitigating multiple descents: A model-agnostic framework for risk monotonization](https://mlsanity.com/paper/2205.12937/) (2022)
- [Measuring the Algorithmic Convergence of Randomized Ensembles: The Regression Setting](https://mlsanity.com/paper/1908.01251/) (2019)
- [Reducing Sampling Ratios Improves Bagging in Sparse Regression](https://mlsanity.com/paper/1812.08808/) (2018)

## Built on by (most cited first, 4 of 4)

- [Corrected generalized cross-validation for finite ensembles of penalized estimators](https://mlsanity.com/paper/2310.01374/) (2023)
- [Implicit Regularization Paths of Weighted Neural Representations](https://mlsanity.com/paper/2408.15784/) (2024)
- [Assumption-Lean Post-Integrated Inference with Surrogate Control Outcomes](https://mlsanity.com/paper/2410.04996/) (2024)
- [Mallows-type model averaging: Non-asymptotic analysis and all-subset combination](https://mlsanity.com/paper/2505.02637/) (2025)

## Abstract (arXiv)

Ensemble methods such as bagging and random forests are ubiquitous in various fields, from finance to genomics. Despite their prevalence, the question of the efficient tuning of ensemble parameters has received relatively little attention. This paper introduces a cross-validation method, ECV (Extrapolated Cross-Validation), for tuning the ensemble and subsample sizes in randomized ensembles. Our method builds on two primary ingredients: initial estimators for small ensemble sizes using out-of-bag errors and a novel risk extrapolation technique that leverages the structure of prediction risk decomposition. By establishing uniform consistency of our risk extrapolation technique over ensemble and subsample sizes, we show that ECV yields δ-optimal (with respect to the oracle-tuned risk) ensembles for squared prediction risk. Our theory accommodates general ensemble predictors, only requires mild moment assumptions, and allows for high-dimensional regimes where the feature dimension grows with the sample size. As a practical case study, we employ ECV to predict surface protein abundances from gene expressions in single-cell multiomics using random forests. In comparison to sample-split cross-validation and K-fold cross-validation, ECV achieves higher accuracy avoiding sample splitting. At the same time, its computational cost is considerably lower owing to the use of the risk extrapolation technique. Additional numerical results validate the finite-sample accuracy of ECV for several common ensemble predictors under a computational constraint on the maximum ensemble size.