# When do Random Forests work?

arXiv: https://arxiv.org/abs/2504.12860 · HTML: https://mlsanity.com/paper/2504.12860/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Builds on (most cited first, 13 of 13)

- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers](https://mlsanity.com/paper/1504.07676/) (2015)
- [To tune or not to tune the number of trees in random forest?](https://mlsanity.com/paper/1705.05654/) (2017)
- [Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success](https://mlsanity.com/paper/1911.00190/) (2019)
- [Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers](https://mlsanity.com/paper/2402.01502/) (2024)
- [Trees, Forests, Chickens, and Eggs: When and Why to Prune Trees in a Random Forest](https://mlsanity.com/paper/2103.16700/) (2021)
- [Estimating the Algorithmic Variance of Randomized Ensembles via the Bootstrap](https://mlsanity.com/paper/1907.08742/) (2019)
- [Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests](https://mlsanity.com/paper/2402.12668/) (2024)
- [On the Pointwise Behavior of Recursive Partitioning and Its Implications for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.10805/) (2022)
- [Optimal Weighted Random Forests](https://mlsanity.com/paper/2305.10042/) (2023)
- [When does Subagging Work?](https://mlsanity.com/paper/2404.01832/) (2024)

## Built on by (most cited first, 2 of 2)

- [Revisiting Randomization in Greedy Model Search](https://mlsanity.com/paper/2506.15643/) (2025)
- [Parallel gradient boosting for flexible estimation of conditional distributions](https://mlsanity.com/paper/2607.13550/) (2026)

## Abstract (arXiv)

We study the effectiveness of randomizing split-directions in random forests. Prior literature has shown that, randomization can reduce variance through decorrelation, and, randomization regularizes and works in low signal-to-noise ratio (SNR) environments. First, we revisit decorrelation and regularization by presenting a systematic analysis of out-of-sample mean-squared error (MSE) for different SNR scenarios based on commonly-used data-generating processes. We point out that there is no consensus in the literature on how the MSE is computed and the terms "variance" and "bias" are not uniquely defined. We obtain MSE decompositions by averaging over training sets first and then covariates. We find that variance reduction tends to increase with the SNR and forests outperform bagging when the SNR is low because, in low SNR cases, variance dominates bias for both methods. Second, we show that the effectiveness of randomization is a question that goes beyond SNRs. We present a simulation study with fixed and moderate SNR, in which we examine the effectiveness of randomization for other data characteristics. In particular, we find that (i) randomization can increase bias in the presence of fat tails in the distribution of covariates; (ii) in the presence of irrelevant covariates randomization is ineffective because bias dominates variance; and (iii) when covariates are mutually correlated randomization tends to be effective because variance dominates bias. Beyond randomization, we find that, for both bagging and random forests, bias can be significantly reduced in the presence of correlated covariates. This last finding goes beyond the prevailing view that averaging mostly works by variance reduction. Given that in practice covariates are often correlated, our findings on correlated covariates could open the way for a better understanding of why random forests work well in many applications.