# Decision Machines: Congruent Decision Trees

arXiv: https://arxiv.org/abs/2101.11347 · HTML: https://mlsanity.com/paper/2101.11347/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 42 references that are in the map.
- First posted on arXiv 27 Jan 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel](https://mlsanity.com/paper/1908.11775/) (2019)
- [Adaptively Sparse Transformers](https://mlsanity.com/paper/1909.00015/) (2019)
- [Deep Neural Decision Trees](https://mlsanity.com/paper/1806.06988/) (2018)
- [A Regularized Framework for Sparse and Structured Neural Attention](https://mlsanity.com/paper/1705.07704/) (2017)
- [On Controllable Sparse Alternatives to Softmax](https://mlsanity.com/paper/1810.11975/) (2018)
- [Neural Random Forests](https://mlsanity.com/paper/1604.07143/) (2016)
- [Sigsoftmax: Reanalysis of the Softmax Bottleneck](https://mlsanity.com/paper/1805.10829/) (2018)
- [Efficient non-greedy optimization of decision trees](https://mlsanity.com/paper/1511.04056/) (2015)
- [Oblique Decision Trees from Derivatives of ReLU Networks](https://mlsanity.com/paper/1909.13488/) (2019)
- [Sparse and Constrained Attention for Neural Machine Translation](https://mlsanity.com/paper/1805.08241/) (2018)

## Abstract (arXiv)

The decision tree recursively partitions the input space into regions and derives axis-aligned decision boundaries from data. Despite its simplicity and interpretability, decision trees lack parameterized representation, which makes it prone to overfitting and difficult to find the optimal structure. We propose Decision Machines, which embed Boolean tests into a binary vector space and represent the tree structure as a matrices, enabling an interleaved traversal of decision trees through matrix computation. Furthermore, we explore the congruence of decision trees and attention mechanisms, opening new avenues for optimizing decision trees and potentially enhancing their predictive power.