# Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers

arXiv: https://arxiv.org/abs/1504.07676 · HTML: https://mlsanity.com/paper/1504.07676/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2015.

## Built on by (most cited first, 30 of 45)

- [Deep learning: a statistical viewpoint](https://mlsanity.com/paper/2103.09177/) (2021)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [How rotational invariance of common kernels prevents generalization in high dimensions](https://mlsanity.com/paper/2104.04244/) (2021)
- [Pushing The Limit of LLM Capacity for Text Classification](https://mlsanity.com/paper/2402.07470/) (2024)
- [Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers](https://mlsanity.com/paper/2402.01502/) (2024)
- [Trees, Forests, Chickens, and Eggs: When and Why to Prune Trees in a Random Forest](https://mlsanity.com/paper/2103.16700/) (2021)
- [Generalized equivalences between subsampling and ridge regularization](https://mlsanity.com/paper/2305.18496/) (2023)
- [Latent Gaussian Model Boosting](https://mlsanity.com/paper/2105.08966/) (2021)
- [The Surprising Harmfulness of Benign Overfitting for Adversarial Robustness](https://mlsanity.com/paper/2401.12236/) (2024)
- [Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond](https://mlsanity.com/paper/2411.00247/) (2024)
- [VisRuler: Visual Analytics for Extracting Decision Rules from Bagged and Boosted Decision Trees](https://mlsanity.com/paper/2112.00334/) (2021)
- [Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning](https://mlsanity.com/paper/2102.12328/) (2021)
- [On the Benefits of Over-parameterization for Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.17592/) (2024)
- [Bit Error and Block Error Rate Training for ML-Assisted Communication](https://mlsanity.com/paper/2210.14103/) (2022)
- [Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests](https://mlsanity.com/paper/2402.12668/) (2024)
- [Classical Statistical (In-Sample) Intuitions Don't Generalize Well: A Note on Bias-Variance Tradeoffs, Overfitting and Moving from Fixed to Random Designs](https://mlsanity.com/paper/2409.18842/) (2024)
- [Classification with Partially Private Features](https://mlsanity.com/paper/2312.07583/) (2023)
- [On the Minimal Error of Empirical Risk Minimization](https://mlsanity.com/paper/2102.12066/) (2021)
- [Scalable Causal Domain Adaptation](https://mlsanity.com/paper/2103.00139/) (2021)
- [Confident Sinkhorn Allocation for Pseudo-Labeling](https://mlsanity.com/paper/2206.05880/) (2022)
- [Ensemble linear interpolators: The role of ensembling](https://mlsanity.com/paper/2309.03354/) (2023)
- [Overview of AdaBoost : Reconciling its views to better understand its dynamics](https://mlsanity.com/paper/2310.18323/) (2023)
- [Yet Another Representation of Binary Decision Trees: A Mathematical Demonstration](https://mlsanity.com/paper/2101.07077/) (2021)
- [AdaBoost and robust one-bit compressed sensing](https://mlsanity.com/paper/2105.02083/) (2021)
- [Anonymous Learning via Look-Alike Clustering: A Precise Analysis of Model Generalization](https://mlsanity.com/paper/2310.04015/) (2023)
- [When do Random Forests work?](https://mlsanity.com/paper/2504.12860/) (2025)
- [Cross-domain error minimization for unsupervised domain adaptation](https://mlsanity.com/paper/2106.15057/) (2021)
- [Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective](https://mlsanity.com/paper/2109.13215/) (2021)
- [VC dimension of partially quantized neural networks in the overparametrized regime](https://mlsanity.com/paper/2110.02456/) (2021)

## Abstract (arXiv)

There is a large literature explaining why AdaBoost is a successful classifier. The literature on AdaBoost focuses on classifier margins and boosting's interpretation as the optimization of an exponential likelihood function. These existing explanations, however, have been pointed out to be incomplete. A random forest is another popular ensemble method for which there is substantially less explanation in the literature. We introduce a novel perspective on AdaBoost and random forests that proposes that the two algorithms work for similar reasons. While both classifiers achieve similar predictive accuracy, random forests cannot be conceived as a direct optimization procedure. Rather, random forests is a self-averaging, interpolating algorithm which creates what we denote as a "spikey-smooth" classifier, and we view AdaBoost in the same light. We conjecture that both AdaBoost and random forests succeed because of this mechanism. We provide a number of examples and some theoretical justification to support this explanation. In the process, we question the conventional wisdom that suggests that boosting algorithms for classification require regularization or early stopping and should be limited to low complexity classes of learners, such as decision stumps. We conclude that boosting should be used like random forests: with large decision trees and without direct regularization or early stopping.