# Benchmarking and Optimization of Gradient Boosting Decision Tree Algorithms

arXiv: https://arxiv.org/abs/1809.04559 · HTML: https://mlsanity.com/paper/1809.04559/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2018.

## Built on by (most cited first, 7 of 7)

- [DANets: Deep Abstract Networks for Tabular Data Classification and Regression](https://mlsanity.com/paper/2112.02962/) (2021)
- [T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interaction](https://mlsanity.com/paper/2211.16887/) (2022)
- [JoinBoost: Grow Trees Over Normalized Data Using Only SQL](https://mlsanity.com/paper/2307.00422/) (2023)
- [Light Gradient Boosting Machine as a Regression Method for Quantitative Structure-Activity Relationships](https://mlsanity.com/paper/2105.08626/) (2021)
- [Large Language Models as Universal Predictors? An Empirical Study on Small Tabular Datasets](https://mlsanity.com/paper/2508.17391/) (2025)
- [Machine Learning For An Explainable Cost Prediction of Medical Insurance](https://mlsanity.com/paper/2311.14139/) (2023)
- [dnamite: A Python Package for Neural Additive Models](https://mlsanity.com/paper/2503.07642/) (2025)

## Abstract (arXiv)

Gradient boosting decision trees (GBDTs) have seen widespread adoption in academia, industry and competitive data science due to their state-of-the-art performance in many machine learning tasks. One relative downside to these models is the large number of hyper-parameters that they expose to the end-user. To maximize the predictive power of GBDT models, one must either manually tune the hyper-parameters, or utilize automated techniques such as those based on Bayesian optimization. Both of these approaches are time-consuming since they involve repeatably training the model for different sets of hyper-parameters. A number of software GBDT packages have started to offer GPU acceleration which can help to alleviate this problem. In this paper, we consider three such packages: XGBoost, LightGBM and Catboost. Firstly, we evaluate the performance of the GPU acceleration provided by these packages using large-scale datasets with varying shapes, sparsities and learning tasks. Then, we compare the packages in the context of hyper-parameter optimization, both in terms of how quickly each package converges to a good validation score, and in terms of generalization performance.