mlsanityData to 7 Oct 2026

AGBoost: Attention-based Modification of Gradient Boosting Machine

arXiv 2207.05724 ↗First posted 12 Jul 2022cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

A new attention-based model for the gradient boosting machine (GBM) called AGBoost (the attention-based gradient boosting) is proposed for solving regression problems. The main idea behind the proposed AGBoost model is to assign attention weights with trainable parameters to iterations of GBM under condition that decision trees are base learners in GBM. Attention weights are determined by applying properties of decision trees and by using the Huber's contamination model which provides an interesting linear dependence between trainable parameters of the attention and the attention weights. This peculiarity allows us to train the attention weights by solving the standard quadratic optimization problem with linear constraints. The attention weights also depend on the discount factor as a tuning parameter, which determines how much the impact of the weight is decreased with the number of iterations. Numerical experiments performed for two types of base learners, original decision trees and extremely randomized trees with various regression datasets illustrate the proposed model.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 35 references that are in the map. First posted on arXiv 12 Jul 2022.

Builds on

19 of 35 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  6. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  7. 2021A Survey of Transformerscited by 399
  8. 2021Random Feature Attentioncited by 350
  9. 2019An Attentive Survey of Attention Modelscited by 123
  10. 2021Dive into Deep Learningcited by 111
  11. 2020Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyondcited by 107
  12. 2021Luna: Linear Unified Nested Attentioncited by 96
7 more
  1. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  2. 2017Deep Forestcited by 46
  3. 2018Multi-Layered Gradient Boosting Decision Treescited by 16
  4. 2022Attention-based Random Forest and Contamination Modelcited by 10
  5. 2021Neural Attention Models in Deep Learning: Survey and Taxonomycited by 8
  6. 2020Soft Gradient Boosting Machinecited by 7
  7. 2020A Generalized Stacking for Implementing Ensembles of Gradient Boosting Machinescited by 3

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 1.

5 papers.

  1. 2022Heterogeneous Treatment Effect with Trained Kernels of the Nadaraya-Watson Regressioncited by 4
  2. 2022Improved Anomaly Detection by Using the Attention-Based Isolation Forestcited by 2
  3. 2022BENK: The Beran Estimator with Neural Kernels for Estimating the Heterogeneous Treatment Effectcited by 1
  4. 2022LARF: Two-level Attention-based Random Forests with a Mixture of Contamination Modelscited by 0
  5. 2023Neural Attention Forests: Transformer-Based Forest Improvementcited by 0