# Randomization as Regularization: A Degrees of Freedom Explanation for Random Forest Success

arXiv: https://arxiv.org/abs/1911.00190 · HTML: https://mlsanity.com/paper/1911.00190/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2019.

## Built on by (most cited first, 30 of 30)

- [Adversarial random forests for density estimation and generative modeling](https://mlsanity.com/paper/2205.09435/) (2022)
- [Large Scale Prediction with Decision Trees](https://mlsanity.com/paper/2104.13881/) (2021)
- [Hierarchical Shrinkage: improving the accuracy and interpretability of tree-based methods](https://mlsanity.com/paper/2202.00858/) (2022)
- [Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers](https://mlsanity.com/paper/2402.01502/) (2024)
- [Trees, Forests, Chickens, and Eggs: When and Why to Prune Trees in a Random Forest](https://mlsanity.com/paper/2103.16700/) (2021)
- [Fast Interpretable Greedy-Tree Sums](https://mlsanity.com/paper/2201.11931/) (2022)
- [Generalized equivalences between subsampling and ridge regularization](https://mlsanity.com/paper/2305.18496/) (2023)
- [Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning](https://mlsanity.com/paper/2102.12328/) (2021)
- [Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests](https://mlsanity.com/paper/2402.12668/) (2024)
- [Muddling Label Regularization: Deep Learning for Tabular Datasets](https://mlsanity.com/paper/2106.04462/) (2021)
- [On the Pointwise Behavior of Recursive Partitioning and Its Implications for Heterogeneous Causal Effect Estimation](https://mlsanity.com/paper/2211.10805/) (2022)
- [Ensemble linear interpolators: The role of ensembling](https://mlsanity.com/paper/2309.03354/) (2023)
- [Exogenous Randomness Empowering Random Forests](https://mlsanity.com/paper/2411.07554/) (2024)
- [When do Random Forests work?](https://mlsanity.com/paper/2504.12860/) (2025)
- [Error Reduction from Stacked Regressions](https://mlsanity.com/paper/2309.09880/) (2023)
- [SR-PredictAO: Session-based Recommendation with High-Capability Predictor Add-On](https://mlsanity.com/paper/2309.12218/) (2023)
- [Understanding overfitting in random forest for probability estimation: a visualization and simulation study](https://mlsanity.com/paper/2402.18612/) (2024)
- [Efficient Decision Trees for Tensor Regressions](https://mlsanity.com/paper/2408.01926/) (2024)
- [Revisiting Randomization in Greedy Model Search](https://mlsanity.com/paper/2506.15643/) (2025)
- [Random Forests as Statistical Procedures: Design, Variance, and Dependence](https://mlsanity.com/paper/2602.13104/) (2026)
- [WildWood: a new Random Forest algorithm](https://mlsanity.com/paper/2109.08010/) (2021)
- [Adaptive Bayesian Sum of Trees Model for Covariate Dependent Spectral Analysis](https://mlsanity.com/paper/2109.14677/) (2021)
- [Local Adaptivity of Gradient Boosting in Histogram Transform Ensemble Learning](https://mlsanity.com/paper/2112.02589/) (2021)
- [Sequential Permutation Testing of Random Forest Variable Importance Measures](https://mlsanity.com/paper/2206.01284/) (2022)
- [TREE: Tree Regularization for Efficient Execution](https://mlsanity.com/paper/2406.12531/) (2024)
- [Global Censored Quantile Random Forest](https://mlsanity.com/paper/2410.12209/) (2024)
- [Targeted tuning of random forests for quantile estimation and prediction intervals](https://mlsanity.com/paper/2507.01430/) (2025)
- [Closed-Form Beta Distribution Estimation from Sparse Statistics with Random Forest Implicit Regularization](https://mlsanity.com/paper/2507.23767/) (2025)
- [Empirical Likelihood for Random Forests and Ensembles](https://mlsanity.com/paper/2511.13934/) (2025)
- [Quantifying and Attributing Submodel Uncertainty in Stochastic Simulation Models and Digital Twins](https://mlsanity.com/paper/2602.16099/) (2026)

## Abstract (arXiv)

Random forests remain among the most popular off-the-shelf supervised machine learning tools with a well-established track record of predictive accuracy in both regression and classification settings. Despite their empirical success as well as a bevy of recent work investigating their statistical properties, a full and satisfying explanation for their success has yet to be put forth. Here we aim to take a step forward in this direction by demonstrating that the additional randomness injected into individual trees serves as a form of implicit regularization, making random forests an ideal model in low signal-to-noise ratio (SNR) settings. Specifically, from a model-complexity perspective, we show that the mtry parameter in random forests serves much the same purpose as the shrinkage penalty in explicitly regularized regression procedures like lasso and ridge regression. To highlight this point, we design a randomized linear-model-based forward selection procedure intended as an analogue to tree-based random forests and demonstrate its surprisingly strong empirical performance. Numerous demonstrations on both real and synthetic data are provided.