# Gradient Boosting Performs Gaussian Process Inference

arXiv: https://arxiv.org/abs/2206.05608 · HTML: https://mlsanity.com/paper/2206.05608/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 11 Jun 2022.

## Builds on (most cited first, 19 of 19)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data](https://mlsanity.com/paper/1808.01204/) (2018)
- [Ensemble Distribution Distillation](https://mlsanity.com/paper/1905.00076/) (2019)
- [Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation](https://mlsanity.com/paper/1902.04760/) (2019)
- [Uncertainty in Gradient Boosting via Ensembles](https://mlsanity.com/paper/2006.10562/) (2020)
- [Faster Convergence of Stochastic Gradient Langevin Dynamics for Non-Log-Concave Sampling](https://mlsanity.com/paper/2010.09597/) (2020)
- [SGLB: Stochastic Gradient Langevin Boosting](https://mlsanity.com/paper/2001.07248/) (2020)
- [The Mondrian Kernel](https://mlsanity.com/paper/1606.05241/) (2016)
- [Minimal Variance Sampling in Stochastic Gradient Boosting](https://mlsanity.com/paper/1910.13204/) (2019)
- [A Neural Tangent Kernel Perspective of Infinite Tree Ensembles](https://mlsanity.com/paper/2109.04983/) (2021)

## Built on by (most cited first, 4 of 4)

- [Reliable and scalable variable importance estimation via warm-start and early stopping](https://mlsanity.com/paper/2412.01120/) (2024)
- [Statistical Inference for Gradient Boosting Regression](https://mlsanity.com/paper/2509.23127/) (2025)
- [Universal Feature Selection for Simultaneous Interpretability of Multitask Datasets](https://mlsanity.com/paper/2403.14466/) (2024)
- [Statistical Inference for Explainable Boosting Machines](https://mlsanity.com/paper/2601.18857/) (2026)

## Abstract (arXiv)

This paper shows that gradient boosting based on symmetric decision trees can be equivalently reformulated as a kernel method that converges to the solution of a certain Kernel Ridge Regression problem. Thus, we obtain the convergence to a Gaussian Process' posterior mean, which, in turn, allows us to easily transform gradient boosting into a sampler from the posterior to provide better knowledge uncertainty estimates through Monte-Carlo estimation of the posterior variance. We show that the proposed sampler allows for better knowledge uncertainty estimates leading to improved out-of-domain detection.