# Learning Representations for Axis-Aligned Decision Forests through Input Perturbation

arXiv: https://arxiv.org/abs/2007.14761 · HTML: https://mlsanity.com/paper/2007.14761/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jul 2020.

## Built on by (most cited first, 4 of 4)

- [Yggdrasil Decision Forests: A Fast and Extensible Decision Forests Library](https://mlsanity.com/paper/2212.02934/) (2022)
- [GradTree: Learning Axis-Aligned Decision Trees with Gradient Descent](https://mlsanity.com/paper/2305.03515/) (2023)
- [Testing Closeness of Multivariate Distributions via Ramsey Theory](https://mlsanity.com/paper/2311.13154/) (2023)
- [Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-Tuning](https://mlsanity.com/paper/2105.03791/) (2021)

## Abstract (arXiv)

Axis-aligned decision forests have long been the leading class of machine learning algorithms for modeling tabular data. In many applications of machine learning such as learning-to-rank, decision forests deliver remarkable performance. They also possess other coveted characteristics such as interpretability. Despite their widespread use and rich history, decision forests to date fail to consume raw structured data such as text, or learn effective representations for them, a factor behind the success of deep neural networks in recent years. While there exist methods that construct smoothed decision forests to achieve representation learning, the resulting models are decision forests in name only: They are no longer axis-aligned, use stochastic decisions, or are not interpretable. Furthermore, none of the existing methods are appropriate for problems that require a Transfer Learning treatment. In this work, we present a novel but intuitive proposal to achieve representation learning for decision forests without imposing new restrictions or necessitating structural changes. Our model is simply a decision forest, possibly trained using any forest learning algorithm, atop a deep neural network. By approximating the gradients of the decision forest through input perturbation, a purely analytical procedure, the decision forest directs the neural network to learn or fine-tune representations. Our framework has the advantage that it is applicable to any arbitrary decision forest and that it allows the use of arbitrary deep neural networks for representation learning. We demonstrate the feasibility and effectiveness of our proposal through experiments on synthetic and benchmark classification datasets.