# Attention-based Random Forest and Contamination Model

arXiv: https://arxiv.org/abs/2201.02880 · HTML: https://mlsanity.com/paper/2201.02880/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 8 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [An Attentive Survey of Attention Models](https://mlsanity.com/paper/1904.02874/) (2019)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond](https://mlsanity.com/paper/2004.11154/) (2020)
- [Luna: Linear Unified Nested Attention](https://mlsanity.com/paper/2106.01540/) (2021)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Deep Forest](https://mlsanity.com/paper/1702.08835/) (2017)
- [Hybrid Random Features](https://mlsanity.com/paper/2110.04367/) (2021)
- [Neural Attention Models in Deep Learning: Survey and Taxonomy](https://mlsanity.com/paper/2112.05909/) (2021)
- [A weighted random survival forest](https://mlsanity.com/paper/1901.00213/) (2019)

## Built on by (most cited first, 10 of 10)

- [Robust Statistical Comparison of Random Variables with Locally Varying Scale of Measurement](https://mlsanity.com/paper/2306.12803/) (2023)
- [Statistical Multicriteria Benchmarking via the GSD-Front](https://mlsanity.com/paper/2406.03924/) (2024)
- [AGBoost: Attention-based Modification of Gradient Boosting Machine](https://mlsanity.com/paper/2207.05724/) (2022)
- [Attention and Self-Attention in Random Forests](https://mlsanity.com/paper/2207.04293/) (2022)
- [Improved Anomaly Detection by Using the Attention-Based Isolation Forest](https://mlsanity.com/paper/2210.02558/) (2022)
- [SurvBETA: Ensemble-Based Survival Models Using Beran Estimators and Several Attention Mechanisms](https://mlsanity.com/paper/2412.07638/) (2024)
- [A Multi-Head Attention Soft Random Forest for Interpretable Patient No-Show Prediction](https://mlsanity.com/paper/2505.17344/) (2025)
- [LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Models](https://mlsanity.com/paper/2210.05168/) (2022)
- [Neural Attention Forests: Transformer-Based Forest Improvement](https://mlsanity.com/paper/2304.05980/) (2023)
- [Ensemble-Based Survival Models with the Self-Attended Beran Estimator Predictions](https://mlsanity.com/paper/2506.07933/) (2025)

## Abstract (arXiv)

A new approach called ABRF (the attention-based random forest) and its modifications for applying the attention mechanism to the random forest (RF) for regression and classification are proposed. The main idea behind the proposed ABRF models is to assign attention weights with trainable parameters to decision trees in a specific way. The weights depend on the distance between an instance, which falls into a corresponding leaf of a tree, and instances, which fall in the same leaf. This idea stems from representation of the Nadaraya-Watson kernel regression in the form of a RF. Three modifications of the general approach are proposed. The first one is based on applying the Huber's contamination model and on computing the attention weights by solving quadratic or linear optimization problems. The second and the third modifications use the gradient-based algorithms for computing trainable parameters. Numerical experiments with various regression and classification datasets illustrate the proposed method.