# Accelerated Componentwise Gradient Boosting using Efficient Data Representation and Momentum-based Optimization

arXiv: https://arxiv.org/abs/2110.03513 · HTML: https://mlsanity.com/paper/2110.03513/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 54 references that are in the map.
- First posted on arXiv 7 Oct 2021.

## Builds on (most cited first, 10 of 10)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [InterpretML: A Unified Framework for Machine Learning Interpretability](https://mlsanity.com/paper/1909.09223/) (2019)
- [Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent](https://mlsanity.com/paper/2002.10583/) (2020)
- [OpenML: An R Package to Connect to the Machine Learning Platform OpenML](https://mlsanity.com/paper/1701.01293/) (2017)
- [Probing for sparse and fast variable selection with model-based boosting](https://mlsanity.com/paper/1702.04561/) (2017)
- [Automatic Gradient Boosting](https://mlsanity.com/paper/1807.03873/) (2018)
- [Boosting Factor-Specific Functional Historical Models for the Detection of Synchronisation in Bioelectrical Signals](https://mlsanity.com/paper/1609.06070/) (2016)

## Built on by (most cited first, 2 of 2)

- [Scalable Higher-Order Tensor Product Spline Models](https://mlsanity.com/paper/2402.01090/) (2024)
- [Privacy-Preserving and Lossless Distributed Estimation of High-Dimensional Generalized Additive Mixed Models](https://mlsanity.com/paper/2210.07723/) (2022)

## Abstract (arXiv)

Componentwise boosting (CWB), also known as model-based boosting, is a variant of gradient boosting that builds on additive models as base learners to ensure interpretability. CWB is thus often used in research areas where models are employed as tools to explain relationships in data. One downside of CWB is its computational complexity in terms of memory and runtime. In this paper, we propose two techniques to overcome these issues without losing the properties of CWB: feature discretization of numerical features and incorporating Nesterov momentum into functional gradient descent. As the latter can be prone to early overfitting, we also propose a hybrid approach that prevents a possibly diverging gradient descent routine while ensuring faster convergence. We perform extensive benchmarks on multiple simulated and real-world data sets to demonstrate the improvements in runtime and memory consumption while maintaining state-of-the-art estimation and prediction performance.