# Learning Multi-Layered GBDT Via Back Propagation

arXiv: https://arxiv.org/abs/2109.11863 · HTML: https://mlsanity.com/paper/2109.11863/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 24 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Adaptive Neural Trees](https://mlsanity.com/paper/1807.06699/) (2018)
- [The Tree Ensemble Layer: Differentiability meets Conditional Computation](https://mlsanity.com/paper/2002.07772/) (2020)
- [Boost then Convolve: Gradient Boosting Meets Graph Neural Networks](https://mlsanity.com/paper/2101.08543/) (2021)
- [Understanding Synthetic Gradients and Decoupled Neural Interfaces](https://mlsanity.com/paper/1703.00522/) (2017)
- [Multi-Layered Gradient Boosting Decision Trees](https://mlsanity.com/paper/1806.00007/) (2018)
- [GBDT-MO: Gradient Boosted Decision Trees for Multiple Outputs](https://mlsanity.com/paper/1909.04373/) (2019)
- [Gradient Boosting With Piece-Wise Linear Regression Trees](https://mlsanity.com/paper/1802.05640/) (2018)
- [Soft Gradient Boosting Machine](https://mlsanity.com/paper/2006.04059/) (2020)
- [LinXGBoost: Extension of XGBoost to Generalized Local Linear Models](https://mlsanity.com/paper/1710.03634/) (2017)

## Abstract (arXiv)

Deep neural networks are able to learn multi-layered representation via back propagation (BP). Although the gradient boosting decision tree (GBDT) is effective for modeling tabular data, it is non-differentiable with respect to its input, thus suffering from learning multi-layered representation. In this paper, we propose a framework of learning multi-layered GBDT via BP. We approximate the gradient of GBDT based on linear regression. Specifically, we use linear regression to replace the constant value at each leaf ignoring the contribution of individual samples to the tree structure. In this way, we estimate the gradient for intermediate representations, which facilitates BP for multi-layered GBDT. Experiments show the effectiveness of the proposed method in terms of performance and representation ability. To the best of our knowledge, this is the first work of optimizing multi-layered GBDT via BP. This work provides a new possibility of exploring deep tree based learning and combining GBDT with neural networks.