# Optimization by gradient boosting

arXiv: https://arxiv.org/abs/1707.05023 · HTML: https://mlsanity.com/paper/1707.05023/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2017.

## Built on by (most cited first, 5 of 5)

- [Machine Learning for Variance Reduction in Online Experiments](https://mlsanity.com/paper/2106.07263/) (2021)
- [Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization](https://mlsanity.com/paper/2206.01409/) (2022)
- [Infinitesimal gradient boosting](https://mlsanity.com/paper/2104.13208/) (2021)
- [Constructive Universal Approximation and Sure Convergence for Multi-Layer Neural Networks](https://mlsanity.com/paper/2507.04779/) (2025)
- [Gradient boosting for extremes: sampling theory and application to insurance](https://mlsanity.com/paper/2606.14268/) (2026)

## Abstract (arXiv)

Gradient boosting is a state-of-the-art prediction technique that sequentially produces a model in the form of linear combinations of simple predictors---typically decision trees---by solving an infinite-dimensional convex optimization problem. We provide in the present paper a thorough analysis of two widespread versions of gradient boosting, and introduce a general framework for studying these algorithms from the point of view of functional optimization. We prove their convergence as the number of iterations tends to infinity and highlight the importance of having a strongly convex risk functional to minimize. We also present a reasonable statistical context ensuring consistency properties of the boosting predictors as the sample size grows. In our approach, the optimization procedures are run forever (that is, without resorting to an early stopping strategy), and statistical regularization is basically achieved via an appropriate L2 penalization of the loss and strong convexity arguments.