# Gradient and Newton Boosting for Classification and Regression

arXiv: https://arxiv.org/abs/1808.03064 · HTML: https://mlsanity.com/paper/1808.03064/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2018.

## Built on by (most cited first, 14 of 14)

- [Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression](https://mlsanity.com/paper/2106.01682/) (2021)
- [Latent Gaussian Model Boosting](https://mlsanity.com/paper/2105.08966/) (2021)
- [Distributional Gradient Boosting Machines](https://mlsanity.com/paper/2204.00778/) (2022)
- [TRBoost: A Generic Gradient Boosting Machine based on Trust-region Method](https://mlsanity.com/paper/2209.13791/) (2022)
- [A Spatio-Temporal Machine Learning Model for Mortgage Credit Risk: Default Probabilities and Loan Portfolios](https://mlsanity.com/paper/2410.02846/) (2024)
- [Forecasting with Hyper-Trees](https://mlsanity.com/paper/2405.07836/) (2024)
- [From Point to probabilistic gradient boosting for claim frequency and severity prediction](https://mlsanity.com/paper/2412.14916/) (2024)
- [Selecting Hyperparameters for Tree-Boosting](https://mlsanity.com/paper/2602.05786/) (2026)
- [Experimental Investigation and Evaluation of Model-based Hyperparameter Optimization](https://mlsanity.com/paper/2107.08761/) (2021)
- [Improved prediction rule ensembling through model-based data generation](https://mlsanity.com/paper/2109.13672/) (2021)
- [MFAI: A Scalable Bayesian Matrix Factorization Approach to Leveraging Auxiliary Information](https://mlsanity.com/paper/2303.02566/) (2023)
- [Gradient Regularized Newton Boosting Trees with Global Convergence](https://mlsanity.com/paper/2605.00581/) (2026)
- [A Censored Transformed Model for Proportional Outcomes with Boundary Mass and an Application to Loss Given Default Modeling](https://mlsanity.com/paper/2606.21515/) (2026)
- [Learning Conditional Expectation Operators via Functional Newton Updates](https://mlsanity.com/paper/2609.35598/) (2026)

## Abstract (arXiv)

Boosting algorithms are frequently used in applied data science and in research. To date, the distinction between boosting with either gradient descent or second-order Newton updates is often not made in both applied and methodological research, and it is thus implicitly assumed that the difference is irrelevant. The goal of this article is to clarify this situation. In particular, we present gradient and Newton boosting, as well as a hybrid variant of the two, in a unified framework. We compare these boosting algorithms with trees as base learners using various datasets and loss functions. Our experiments show that Newton boosting outperforms gradient and hybrid gradient-Newton boosting in terms of predictive accuracy on the majority of datasets. We also present evidence that the reason for this is not faster convergence of Newton boosting. In addition, we introduce a novel tuning parameter for tree-based Newton boosting which is interpretable and important for predictive accuracy.