# The Role of Depth, Width, and Tree Size in Expressiveness of Deep Forest

arXiv: https://arxiv.org/abs/2407.05108 · HTML: https://mlsanity.com/paper/2407.05108/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 49 references that are in the map.
- First posted on arXiv 6 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [The Power of Depth for Feedforward Neural Networks](https://mlsanity.com/paper/1512.03965/) (2015)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [Depth Separation for Neural Networks](https://mlsanity.com/paper/1702.08489/) (2017)
- [Deep Forest](https://mlsanity.com/paper/1702.08835/) (2017)
- [Iterative Random Forests to detect predictive and stable high-order interactions](https://mlsanity.com/paper/1706.08457/) (2017)
- [Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron](https://mlsanity.com/paper/2302.10034/) (2023)
- [On the Number of Regions of Piecewise Linear Neural Networks](https://mlsanity.com/paper/2206.08615/) (2022)
- [Narrowing the Gap: Random Forests In Theory and In Practice](https://mlsanity.com/paper/1310.1415/) (2013)
- [Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers](https://mlsanity.com/paper/2402.01502/) (2024)
- [Multi-Label Learning with Deep Forest](https://mlsanity.com/paper/1911.06557/) (2019)
- [Width is Less Important than Depth in ReLU Neural Networks](https://mlsanity.com/paper/2202.03841/) (2022)
- [Distributed Deep Forest and its Application to Automatic Detection of Cash-out Fraud](https://mlsanity.com/paper/1805.04234/) (2018)
- [Decision tree heuristics can fail, even in the smoothed setting](https://mlsanity.com/paper/2107.00819/) (2021)
- [Towards Understanding Theoretical Advantages of Complex-Reaction Networks](https://mlsanity.com/paper/2108.06711/) (2021)
- [Analyzing the tree-layer structure of Deep Forests](https://mlsanity.com/paper/2010.15690/) (2020)
- [On the Convergence of CART under Sufficient Impurity Decrease Condition](https://mlsanity.com/paper/2310.17114/) (2023)
- [Stability of Random Forests and Coverage of Random-Forest Prediction Intervals](https://mlsanity.com/paper/2310.18814/) (2023)
- [Discriminative Metric Learning with Deep Forest](https://mlsanity.com/paper/1705.09620/) (2017)
- [Theoretical Exploration of Flexible Transmitter Model](https://mlsanity.com/paper/2111.06027/) (2021)

## Abstract (arXiv)

Random forests are classical ensemble algorithms that construct multiple randomized decision trees and aggregate their predictions using naive averaging. citetzhou2019deep further propose a deep forest algorithm with multi-layer forests, which outperforms random forests in various tasks. The performance of deep forests is related to three hyperparameters in practice: depth, width, and tree size, but little has been known about its theoretical explanation. This work provides the first upper and lower bounds on the approximation complexity of deep forests concerning the three hyperparameters. Our results confirm the distinctive role of depth, which can exponentially enhance the expressiveness of deep forests compared with width and tree size. Experiments confirm the theoretical findings.