# To Bag is to Prune

arXiv: https://arxiv.org/abs/2008.07063 · HTML: https://mlsanity.com/paper/2008.07063/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Aug 2020.

## Built on by (most cited first, 5 of 5)

- [Maximally Machine-Learnable Portfolios](https://mlsanity.com/paper/2306.05568/) (2023)
- [SETAR-Tree: A Novel and Accurate Tree Algorithm for Global Time Series Forecasting](https://mlsanity.com/paper/2211.08661/) (2022)
- [Slow-Growing Trees](https://mlsanity.com/paper/2103.01926/) (2021)
- [Ordinary Least Squares as an Attention Mechanism](https://mlsanity.com/paper/2504.09663/) (2025)
- [Quantifying the Risk-Return Tradeoff in Forecasting](https://mlsanity.com/paper/2605.09712/) (2026)

## Abstract (arXiv)

It is notoriously difficult to build a bad Random Forest (RF). Concurrently, RF blatantly overfits in-sample without any apparent consequence out-of-sample. Standard arguments, like the classic bias-variance trade-off or double descent, cannot rationalize this paradox. I propose a new explanation: bootstrap aggregation and model perturbation as implemented by RF automatically prune a latent "true" tree. More generally, randomized ensembles of greedily optimized learners implicitly perform optimal early stopping out-of-sample. So there is no need to tune the stopping point. By construction, novel variants of Boosting and MARS are also eligible for automatic tuning. I empirically demonstrate the property, with simulated and real data, by reporting that these new completely overfitting ensembles perform similarly to their tuned counterparts -- or better.