# Uncertainty in Gradient Boosting via Ensembles

arXiv: https://arxiv.org/abs/2006.10562 · HTML: https://mlsanity.com/paper/2006.10562/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 30 of 56)

- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [Uncertainty quantification in fine-tuned LLMs using LoRA ensembles](https://mlsanity.com/paper/2402.12264/) (2024)
- [Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning](https://mlsanity.com/paper/2107.10663/) (2021)
- [Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean](https://mlsanity.com/paper/2301.11862/) (2023)
- [Adapting and Evaluating Influence-Estimation Methods for Gradient-Boosted Decision Trees](https://mlsanity.com/paper/2205.00359/) (2022)
- [When to intervene? Prescriptive Process Monitoring Under Uncertainty and Resource Constraints](https://mlsanity.com/paper/2206.07745/) (2022)
- [FSP-Laplace: Function-Space Priors for the Laplace Approximation in Bayesian Deep Learning](https://mlsanity.com/paper/2407.13711/) (2024)
- [Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees](https://mlsanity.com/paper/2205.11412/) (2022)
- [On the Calibration of Probabilistic Classifier Sets](https://mlsanity.com/paper/2205.10082/) (2022)
- [SeqNAS: Neural Architecture Search for Event Sequence Classification](https://mlsanity.com/paper/2401.03246/) (2024)
- [Uncertainty-Aware Mixed-Variable Machine Learning for Materials Design](https://mlsanity.com/paper/2207.04994/) (2022)
- [Benchmarking Machine Learning Robustness in Covid-19 Genome Sequence Classification](https://mlsanity.com/paper/2207.08898/) (2022)
- [Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs](https://mlsanity.com/paper/2410.06431/) (2024)
- [Uncertainty-Aware Boosted Ensembling in Multi-Modal Settings](https://mlsanity.com/paper/2104.10715/) (2021)
- [On Uncertainty Estimation by Tree-based Surrogate Models in Sequential Model-based Optimization](https://mlsanity.com/paper/2202.10669/) (2022)
- [Gradient Boosting Performs Gaussian Process Inference](https://mlsanity.com/paper/2206.05608/) (2022)
- [Prescriptive Process Monitoring Under Resource Constraints: A Reinforcement Learning Approach](https://mlsanity.com/paper/2307.06564/) (2023)
- [MorphBoost: Self-Organizing Universal Gradient Boosting with Adaptive Tree Morphing](https://mlsanity.com/paper/2511.13234/) (2025)
- [Confident Sinkhorn Allocation for Pseudo-Labeling](https://mlsanity.com/paper/2206.05880/) (2022)
- [Regularized KL-Divergence for Well-Defined Function-Space Variational Inference in Bayesian neural networks](https://mlsanity.com/paper/2406.04317/) (2024)
- [Evaluating Predictive Uncertainty and Robustness to Distributional Shift Using Real World Data](https://mlsanity.com/paper/2111.04665/) (2021)
- [Towards Clear Expectations for Uncertainty Estimation](https://mlsanity.com/paper/2207.13341/) (2022)
- [Surrogate uncertainty estimation for your time series forecasting black-box: learn when to trust](https://mlsanity.com/paper/2302.02834/) (2023)
- [Machine-Learning Solutions for the Analysis of Single-Particle Diffusion Trajectories](https://mlsanity.com/paper/2308.09414/) (2023)
- [SEMF: Supervised Expectation-Maximization Framework for Predicting Intervals](https://mlsanity.com/paper/2405.18176/) (2024)
- [Diffusion Boosted Trees](https://mlsanity.com/paper/2406.01813/) (2024)
- [Advancing Molecular Machine Learning Representations with Stereoelectronics-Infused Molecular Graphs](https://mlsanity.com/paper/2408.04520/) (2024)
- [How does Bayesian Sampling help Membership Inference Attacks?](https://mlsanity.com/paper/2503.07482/) (2025)
- [TabResFlow: A Normalizing Spline Flow Model for Probabilistic Univariate Tabular Regression](https://mlsanity.com/paper/2508.17056/) (2025)

## Abstract (arXiv)

For many practical, high-risk applications, it is essential to quantify uncertainty in a model's predictions to avoid costly mistakes. While predictive uncertainty is widely studied for neural networks, the topic seems to be under-explored for models based on gradient boosting. However, gradient boosting often achieves state-of-the-art results on tabular data. This work examines a probabilistic ensemble-based framework for deriving uncertainty estimates in the predictions of gradient boosting classification and regression models. We conducted experiments on a range of synthetic and real datasets and investigated the applicability of ensemble approaches to gradient boosting models that are themselves ensembles of decision trees. Our analysis shows that ensembles of gradient boosting models successfully detect anomalous inputs while having limited ability to improve the predicted total uncertainty. Importantly, we also propose a concept of a virtual ensemble to get the benefits of an ensemble via only one gradient boosting model, which significantly reduces complexity.