# Probabilistic Gradient Boosting Machines for Large-Scale Probabilistic Regression

arXiv: https://arxiv.org/abs/2106.01682 · HTML: https://mlsanity.com/paper/2106.01682/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 24 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 9 of 9)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/1905.03806/) (2019)
- [NGBoost: Natural Gradient Boosting for Probabilistic Prediction](https://mlsanity.com/paper/1910.03225/) (2019)
- [Gradient and Newton Boosting for Classification and Regression](https://mlsanity.com/paper/1808.03064/) (2018)
- [Stochastic Gradient Trees](https://mlsanity.com/paper/1901.07777/) (2019)

## Built on by (most cited first, 20 of 20)

- [A review of predictive uncertainty estimation with machine learning](https://mlsanity.com/paper/2209.08307/) (2022)
- [Transformer Uncertainty Estimation with Hierarchical Stochastic Attention](https://mlsanity.com/paper/2112.13776/) (2021)
- [Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees](https://mlsanity.com/paper/2205.11412/) (2022)
- [Distributional Gradient Boosting Machines](https://mlsanity.com/paper/2204.00778/) (2022)
- [Treeffuser: Probabilistic Predictions via Conditional Diffusions with Gradient-Boosted Trees](https://mlsanity.com/paper/2406.07658/) (2024)
- [Hierarchical Forecasting at Scale](https://mlsanity.com/paper/2310.12809/) (2023)
- [Unmasking Trees for Tabular Data](https://mlsanity.com/paper/2407.05593/) (2024)
- [Sample-based Uncertainty Quantification with a Single Deterministic Neural Network](https://mlsanity.com/paper/2209.08418/) (2022)
- [TabResFlow: A Normalizing Spline Flow Model for Probabilistic Univariate Tabular Regression](https://mlsanity.com/paper/2508.17056/) (2025)
- [TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression](https://mlsanity.com/paper/2206.04140/) (2022)
- [ProbSAINT: Probabilistic Tabular Regression for Used Car Pricing](https://mlsanity.com/paper/2403.03812/) (2024)
- [Forecasting with Hyper-Trees](https://mlsanity.com/paper/2405.07836/) (2024)
- [From Point to probabilistic gradient boosting for claim frequency and severity prediction](https://mlsanity.com/paper/2412.14916/) (2024)
- [High-Order Optimization of Gradient Boosted Decision Trees](https://mlsanity.com/paper/2211.11367/) (2022)
- [Embracing Uncertainty Flexibility: Harnessing a Supervised Tree Kernel to Empower Ensemble Modelling for 2D Echocardiography-Based Prediction of Right Ventricular Volume](https://mlsanity.com/paper/2403.03229/) (2024)
- [Generative modeling of density regression through tree flows](https://mlsanity.com/paper/2406.05260/) (2024)
- [News and Load: Social and Economic Drivers of Regional Multi-horizon Electricity Demand Forecasting](https://mlsanity.com/paper/2406.06641/) (2024)
- [Centrum: Model-based Database Auto-tuning with Minimal Distributional Assumptions](https://mlsanity.com/paper/2510.22734/) (2025)
- [Probabilistic Wind Power Forecasting with Tree-Based Machine Learning and Weather Ensembles](https://mlsanity.com/paper/2602.13010/) (2026)
- [LoBoost: Fast Model-Native Local Conformal Prediction for Gradient-Boosted Trees](https://mlsanity.com/paper/2602.22432/) (2026)

## Abstract (arXiv)

Gradient Boosting Machines (GBM) are hugely popular for solving tabular data problems. However, practitioners are not only interested in point predictions, but also in probabilistic predictions in order to quantify the uncertainty of the predictions. Creating such probabilistic predictions is difficult with existing GBM-based solutions: they either require training multiple models or they become too computationally expensive to be useful for large-scale settings. We propose Probabilistic Gradient Boosting Machines (PGBM), a method to create probabilistic predictions with a single ensemble of decision trees in a computationally efficient manner. PGBM approximates the leaf weights in a decision tree as a random variable, and approximates the mean and variance of each sample in a dataset via stochastic tree ensemble update equations. These learned moments allow us to subsequently sample from a specified distribution after training. We empirically demonstrate the advantages of PGBM compared to existing state-of-the-art methods: (i) PGBM enables probabilistic estimates without compromising on point performance in a single model, (ii) PGBM learns probabilistic estimates via a single model only (and without requiring multi-parameter boosting), and thereby offers a speedup of up to several orders of magnitude over existing state-of-the-art methods on large datasets, and (iii) PGBM achieves accurate probabilistic estimates in tasks with complex differentiable loss functions, such as hierarchical time series problems, where we observed up to 10% improvement in point forecasting performance and up to 300% improvement in probabilistic forecasting performance.