# Can a Single Tree Outperform an Entire Forest?

arXiv: https://arxiv.org/abs/2411.17003 · HTML: https://mlsanity.com/paper/2411.17003/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Hyperparameters and Tuning Strategies for Random Forest](https://mlsanity.com/paper/1804.03515/) (2018)
- [Deep Neural Decision Trees](https://mlsanity.com/paper/1806.06988/) (2018)
- [NBDT: Neural-Backed Decision Trees](https://mlsanity.com/paper/2004.00221/) (2020)
- [To tune or not to tune the number of trees in random forest?](https://mlsanity.com/paper/1705.05654/) (2017)
- [A Scalable MIP-based Method for Learning Optimal Multivariate Decision Trees](https://mlsanity.com/paper/2011.03375/) (2020)
- [Learning Accurate Decision Trees with Bandit Feedback via Quantized Gradient Descent](https://mlsanity.com/paper/2102.07567/) (2021)
- [HHCART: An Oblique Decision Tree](https://mlsanity.com/paper/1504.03415/) (2015)
- [End-to-end Learning of Deterministic Decision Trees](https://mlsanity.com/paper/1712.02743/) (2017)

## Built on by (most cited first, 1 of 1)

- [A PyTorch-Enabled Tool for Synthetic Event Camera Data Generation and Algorithm Development](https://mlsanity.com/paper/2503.09754/) (2025)

## Abstract (arXiv)

The prevailing mindset is that a single decision tree underperforms classic random forests in testing accuracy, despite its advantages in interpretability and lightweight structure. This study challenges such a mindset by significantly improving the testing accuracy of an oblique regression tree through our gradient-based entire tree optimization framework, making its performance comparable to the classic random forest. Our approach reformulates tree training as a differentiable unconstrained optimization task, employing a scaled sigmoid approximation strategy. To ameliorate numerical instability, we propose an algorithmic scheme that solves a sequence of increasingly accurate approximations. Additionally, a subtree polish strategy is implemented to reduce approximation errors accumulated across the tree. Extensive experiments on 16 datasets demonstrate that our optimized tree outperforms the classic random forest by an average of 2.03% improvements in testing accuracy.