# Trees, Forests, Chickens, and Eggs: When and Why to Prune Trees in a Random Forest

arXiv: https://arxiv.org/abs/2103.16700 · HTML: https://mlsanity.com/paper/2103.16700/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 46 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [A Random Forest Guided Tour](https://mlsanity.com/paper/1511.05741/) (2015)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [Hyperparameters and Tuning Strategies for Random Forest](https://mlsanity.com/paper/1804.03515/) (2018)
- [Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers](https://mlsanity.com/paper/1504.07676/) (2015)
- [Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success](https://mlsanity.com/paper/1911.00190/) (2019)
- [The Implicit Regularization of Ordinary Least Squares Ensembles](https://mlsanity.com/paper/1910.04743/) (2019)
- [Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance](https://mlsanity.com/paper/2003.03629/) (2020)

## Built on by (most cited first, 13 of 13)

- [Large Scale Prediction with Decision Trees](https://mlsanity.com/paper/2104.13881/) (2021)
- [Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers](https://mlsanity.com/paper/2402.01502/) (2024)
- [Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests](https://mlsanity.com/paper/2402.12668/) (2024)
- [Alpha-Trimming: Locally Adaptive Tree Pruning for Random Forests](https://mlsanity.com/paper/2408.07151/) (2024)
- [When do Random Forests work?](https://mlsanity.com/paper/2504.12860/) (2025)
- [When does Subagging Work?](https://mlsanity.com/paper/2404.01832/) (2024)
- [WildWood: a new Random Forest algorithm](https://mlsanity.com/paper/2109.08010/) (2021)
- [Combining Predictions under Uncertainty: The Case of Random Decision Trees](https://mlsanity.com/paper/2208.07403/) (2022)
- [Red Teaming Models for Hyperspectral Image Analysis Using Explainable AI](https://mlsanity.com/paper/2403.08017/) (2024)
- [Global Censored Quantile Random Forest](https://mlsanity.com/paper/2410.12209/) (2024)
- [Analyzing decision tree bias towards the minority class](https://mlsanity.com/paper/2501.04903/) (2025)
- [Diversity Conscious Refined Random Forest](https://mlsanity.com/paper/2507.00467/) (2025)
- [Targeted tuning of random forests for quantile estimation and prediction intervals](https://mlsanity.com/paper/2507.01430/) (2025)

## Abstract (arXiv)

Due to their long-standing reputation as excellent off-the-shelf predictors, random forests continue remain a go-to model of choice for applied statisticians and data scientists. Despite their widespread use, however, until recently, little was known about their inner-workings and about which aspects of the procedure were driving their success. Very recently, two competing hypotheses have emerged -- one based on interpolation and the other based on regularization. This work argues in favor of the latter by utilizing the regularization framework to reexamine the decades-old question of whether individual trees in an ensemble ought to be pruned. Despite the fact that default constructions of random forests use near full depth trees in most popular software packages, here we provide strong evidence that tree depth should be seen as a natural form of regularization across the entire procedure. In particular, our work suggests that random forests with shallow trees are advantageous when the signal-to-noise ratio in the data is low. In building up this argument, we also critique the newly popular notion of "double descent" in random forests by drawing parallels to U-statistics and arguing that the noticeable jumps in random forest accuracy are the result of simple averaging rather than interpolation.