# Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers

arXiv: https://arxiv.org/abs/2402.01502 · HTML: https://mlsanity.com/paper/2402.01502/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 49 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [A Random Forest Guided Tour](https://mlsanity.com/paper/1511.05741/) (2015)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [Classification vs regression in overparameterized regimes: Does the loss function matter?](https://mlsanity.com/paper/2005.08054/) (2020)
- [Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers](https://mlsanity.com/paper/1504.07676/) (2015)
- [Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife](https://mlsanity.com/paper/1311.4555/) (2013)
- [A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning](https://mlsanity.com/paper/2310.18988/) (2023)
- [Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success](https://mlsanity.com/paper/1911.00190/) (2019)
- [The Random Forest Kernel and other kernels for big data from random partitions](https://mlsanity.com/paper/1402.4293/) (2014)
- [Trees, Forests, Chickens, and Eggs: When and Why to Prune Trees in a Random Forest](https://mlsanity.com/paper/2103.16700/) (2021)
- [From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation](https://mlsanity.com/paper/1704.08160/) (2017)
- [Boosting Random Forests to Reduce Bias; One-Step Boosted Forest and its Variance Estimate](https://mlsanity.com/paper/1803.08000/) (2018)
- [Measuring the Algorithmic Convergence of Randomized Ensembles: The Regression Setting](https://mlsanity.com/paper/1908.01251/) (2019)
- [Making Sense of Random Forest Probabilities: a Kernel Perspective](https://mlsanity.com/paper/1812.05792/) (2018)

## Built on by (most cited first, 15 of 15)

- [Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond](https://mlsanity.com/paper/2411.00247/) (2024)
- [Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs](https://mlsanity.com/paper/2409.18842/) (2024)
- [Not All Explanations for Deep Learning Phenomena Are Equally Valuable](https://mlsanity.com/paper/2506.23286/) (2025)
- [Exogenous Randomness Empowering Random Forests](https://mlsanity.com/paper/2411.07554/) (2024)
- [When do Random Forests work?](https://mlsanity.com/paper/2504.12860/) (2025)
- [When does Subagging Work?](https://mlsanity.com/paper/2404.01832/) (2024)
- [Revisiting Randomization in Greedy Model Search](https://mlsanity.com/paper/2506.15643/) (2025)
- [Regularizing Extrapolation in Causal Inference](https://mlsanity.com/paper/2509.17180/) (2025)
- [Random Forests as Statistical Procedures: Design, Variance, and Dependence](https://mlsanity.com/paper/2602.13104/) (2026)
- [The Role of Depth, Width, and Tree Size in Expressiveness of Deep Forest](https://mlsanity.com/paper/2407.05108/) (2024)
- [Prediction Models That Learn to Avoid Missing Values](https://mlsanity.com/paper/2505.03393/) (2025)
- [Exact Finite-Sample Variance Decomposition of Subagging: A Spectral Filtering Perspective](https://mlsanity.com/paper/2604.10469/) (2026)
- [A Rigorous, Tractable Measure of Model Complexity](https://mlsanity.com/paper/2605.21167/) (2026)
- [CART Random Forests as Sequential Allocation over Random Opportunity Sets: A Stochastic-Control Theory of Ensemble Risk](https://mlsanity.com/paper/2605.26675/) (2026)
- [Feature Bagging Provides Stability](https://mlsanity.com/paper/2607.26964/) (2026)

## Abstract (arXiv)

Despite their remarkable effectiveness and broad application, the drivers of success underlying ensembles of trees are still not fully understood. In this paper, we highlight how interpreting tree ensembles as adaptive and self-regularizing smoothers can provide new intuition and deeper insight to this topic. We use this perspective to show that, when studied as smoothers, randomized tree ensembles not only make predictions that are quantifiably more smooth than the predictions of the individual trees they consist of, but also further regulate their smoothness at test-time based on the dissimilarity between testing and training inputs. First, we use this insight to revisit, refine and reconcile two recent explanations of forest success by providing a new way of quantifying the conjectured behaviors of tree ensembles objectively by measuring the effective degree of smoothing they imply. Then, we move beyond existing explanations for the mechanisms by which tree ensembles improve upon individual trees and challenge the popular wisdom that the superior performance of forests should be understood as a consequence of variance reduction alone. We argue that the current high-level dichotomy into bias- and variance-reduction prevalent in statistics is insufficient to understand tree ensembles -- because the prevailing definition of bias does not capture differences in the expressivity of the hypothesis classes formed by trees and forests. Instead, we show that forests can improve upon trees by three distinct mechanisms that are usually implicitly entangled. In particular, we demonstrate that the smoothing effect of ensembling can reduce variance in predictions due to noise in outcome generation, reduce variability in the quality of the learned function given fixed input data and reduce potential bias in learnable functions by enriching the available hypothesis space.