# Dive into Decision Trees and Forests: A Theoretical Demonstration

arXiv: https://arxiv.org/abs/2101.08656 · HTML: https://mlsanity.com/paper/2101.08656/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 97 references that are in the map.
- First posted on arXiv 20 Jan 2021.

## Builds on (most cited first, 19 of 19)

- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [Optimal Sparse Decision Trees](https://mlsanity.com/paper/1904.12847/) (2019)
- [Generalized and Scalable Optimal Sparse Decision Trees](https://mlsanity.com/paper/2006.08690/) (2020)
- [Deep Neural Decision Trees](https://mlsanity.com/paper/1806.06988/) (2018)
- [Adaptive Neural Trees](https://mlsanity.com/paper/1807.06699/) (2018)
- [The Tree Ensemble Layer: Differentiability meets Conditional Computation](https://mlsanity.com/paper/2002.07772/) (2020)
- [NBDT: Neural-Backed Decision Trees](https://mlsanity.com/paper/2004.00221/) (2020)
- [Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers](https://mlsanity.com/paper/1504.07676/) (2015)
- [Bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects](https://mlsanity.com/paper/1706.09523/) (2017)
- [Fair Forests: Regularized Tree Induction to Minimize Model Bias](https://mlsanity.com/paper/1712.08197/) (2017)
- [Neural Decision Trees](https://mlsanity.com/paper/1702.07360/) (2017)
- [Sparse Projection Oblique Randomer Forests](https://mlsanity.com/paper/1506.03410/) (2015)
- [Decision Forests, Convolutional Networks and the Models in-Between](https://mlsanity.com/paper/1603.01250/) (2016)
- [Sparsity in Optimal Randomized Classification Trees](https://mlsanity.com/paper/2002.09191/) (2020)
- [Optimal Generalized Decision Trees via Integer Programming](https://mlsanity.com/paper/1612.03225/) (2016)
- [Bayesian Additive Regression Trees using Bayesian Model Averaging](https://mlsanity.com/paper/1507.00181/) (2015)
- [The Power of Unbiased Recursive Partitioning: A Unifying View of CTree, MOB, and GUIDE](https://mlsanity.com/paper/1906.10179/) (2019)
- [Column generation based math-heuristic for classification trees](https://mlsanity.com/paper/1810.06684/) (2018)
- [A Bayesian Decision Tree Algorithm](https://mlsanity.com/paper/1901.03214/) (2019)

## Abstract (arXiv)

Based on decision trees, many fields have arguably made tremendous progress in recent years. In simple words, decision trees use the strategy of "divide-and-conquer" to divide the complex problem on the dependency between input features and labels into smaller ones. While decision trees have a long history, recent advances have greatly improved their performance in computational advertising, recommender system, information retrieval, etc. We introduce common tree-based models (e.g., Bayesian CART, Bayesian regression splines) and training techniques (e.g., mixed integer programming, alternating optimization, gradient descent). Along the way, we highlight probabilistic characteristics of tree-based models and explain their practical and theoretical benefits. Except machine learning and data mining, we try to show theoretical advances on tree-based models from other fields such as statistics and operation research. We list the reproducible resource at the end of each method.