# Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees

arXiv: https://arxiv.org/abs/2205.11412 · HTML: https://mlsanity.com/paper/2205.11412/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 89 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 21 of 21)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Conformalized Quantile Regression](https://mlsanity.com/paper/1905.03222/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/1905.03806/) (2019)
- [NGBoost: Natural Gradient Boosting for Probabilistic Prediction](https://mlsanity.com/paper/1910.03225/) (2019)
- [Deep Factors for Forecasting](https://mlsanity.com/paper/1905.12417/) (2019)
- [Model Agnostic Supervised Local Explanations](https://mlsanity.com/paper/1807.02910/) (2018)
- [Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification](https://mlsanity.com/paper/2109.10254/) (2021)
- [Uncertainty in Gradient Boosting via Ensembles](https://mlsanity.com/paper/2006.10562/) (2020)
- [Improving Palliative Care with Deep Learning](https://mlsanity.com/paper/1711.06402/) (2017)
- [Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression](https://mlsanity.com/paper/2106.01682/) (2021)
- [Distributional Regression Forests for Probabilistic Precipitation Forecasting in Complex Terrain](https://mlsanity.com/paper/1804.02921/) (2018)
- [A Tale Of Two Long Tails](https://mlsanity.com/paper/2107.13098/) (2021)
- [The Random Forest Kernel and other kernels for big data from random partitions](https://mlsanity.com/paper/1402.4293/) (2014)
- [SGLB: Stochastic Gradient Langevin Boosting](https://mlsanity.com/paper/2001.07248/) (2020)
- [TREX: Tree-Ensemble Representer-Point Explanations](https://mlsanity.com/paper/2009.05530/) (2020)

## Built on by (most cited first, 9 of 9)

- [Treeffuser: Probabilistic Predictions via Conditional Diffusions with Gradient-Boosted Trees](https://mlsanity.com/paper/2406.07658/) (2024)
- [Unmasking Trees for Tabular Data](https://mlsanity.com/paper/2407.05593/) (2024)
- [Sample-based Uncertainty Quantification with a Single Deterministic Neural Network](https://mlsanity.com/paper/2209.08418/) (2022)
- [Statistical Inference for Gradient Boosting Regression](https://mlsanity.com/paper/2509.23127/) (2025)
- [When Active Learning Fails, Uncalibrated Out of Distribution Uncertainty Quantification Might Be the Problem](https://mlsanity.com/paper/2511.17760/) (2025)
- [Model Uncertainty based Active Learning on Tabular Data using Boosted Trees](https://mlsanity.com/paper/2310.19573/) (2023)
- [Embracing Uncertainty Flexibility: Harnessing a Supervised Tree Kernel to Empower Ensemble Modelling for 2D Echocardiography-Based Prediction of Right Ventricular Volume](https://mlsanity.com/paper/2403.03229/) (2024)
- [Barlow Twins Deep Neural Network for Advanced 1D Drug-Target Interaction Prediction](https://mlsanity.com/paper/2408.00040/) (2024)
- [Conditioning Tree-Based Diffusions and Flows for Probabilistic Tabular Regression](https://mlsanity.com/paper/2607.28864/) (2026)

## Abstract (arXiv)

Gradient-boosted regression trees (GBRTs) are hugely popular for solving tabular regression problems, but provide no estimate of uncertainty. We propose Instance-Based Uncertainty estimation for Gradient-boosted regression trees (IBUG), a simple method for extending any GBRT point predictor to produce probabilistic predictions. IBUG computes a non-parametric distribution around a prediction using the k-nearest training instances, where distance is measured with a tree-ensemble kernel. The runtime of IBUG depends on the number of training examples at each leaf in the ensemble, and can be improved by sampling trees or training instances. Empirically, we find that IBUG achieves similar or better performance than the previous state-of-the-art across 22 benchmark regression datasets. We also find that IBUG can achieve improved probabilistic performance by using different base GBRT models, and can more flexibly model the posterior distribution of a prediction than competing methods. We also find that previous methods suffer from poor probabilistic calibration on some datasets, which can be mitigated using a scalar factor tuned on the validation data. Source code is available at https://www.github.com/jjbrophy47/ibug.