# Statistical Inference for Gradient Boosting Regression

arXiv: https://arxiv.org/abs/2509.23127 · HTML: https://mlsanity.com/paper/2509.23127/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 27 Sep 2025.

## Builds on (most cited first, 19 of 19)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [Generalized Random Forests](https://mlsanity.com/paper/1610.01271/) (2016)
- [NGBoost: Natural Gradient Boosting for Probabilistic Prediction](https://mlsanity.com/paper/1910.03225/) (2019)
- [Uncertainty in Gradient Boosting via Ensembles](https://mlsanity.com/paper/2006.10562/) (2020)
- [Confidence Intervals for Random Forests: The Jackknife and the Infinitesimal Jackknife](https://mlsanity.com/paper/1311.4555/) (2013)
- [Local Linear Forests](https://mlsanity.com/paper/1807.11408/) (2018)
- [DART: Dropouts meet Multiple Additive Regression Trees](https://mlsanity.com/paper/1505.01866/) (2015)
- [Instance-Based Uncertainty Estimation for Gradient-Boosted Regression Trees](https://mlsanity.com/paper/2205.11412/) (2022)
- [Scalable and Efficient Hypothesis Testing with Random Forests](https://mlsanity.com/paper/1904.07830/) (2019)
- [The Impossibility of Parallelizing Boosting](https://mlsanity.com/paper/2301.09627/) (2023)
- [Gradient Boosting Performs Gaussian Process Inference](https://mlsanity.com/paper/2206.05608/) (2022)
- [Revisiting Structured Dropout](https://mlsanity.com/paper/2210.02570/) (2022)
- [Quantile Extreme Gradient Boosting for Uncertainty Quantification](https://mlsanity.com/paper/2304.11732/) (2023)
- [Optimal Parallelization of Boosting](https://mlsanity.com/paper/2408.16653/) (2024)

## Built on by (most cited first, 2 of 2)

- [Statistical Inference for Explainable Boosting Machines](https://mlsanity.com/paper/2601.18857/) (2026)
- [LoBoost: Fast Model-Native Local Conformal Prediction for Gradient-Boosted Trees](https://mlsanity.com/paper/2602.22432/) (2026)

## Abstract (arXiv)

Gradient boosting is widely popular due to its flexibility and predictive accuracy. However, statistical inference and uncertainty quantification for gradient boosting remain challenging and under-explored. We propose a unified framework for statistical inference in gradient boosting regression. Our framework integrates dropout or parallel training with a recently proposed regularization procedure that allows for a central limit theorem (CLT) for boosting. With these enhancements, we surprisingly find that increasing the dropout rate and the number of trees grown in parallel at each iteration substantially enhances signal recovery and overall performance. Our resulting algorithms enjoy similar CLTs, which we use to construct built-in confidence intervals, prediction intervals, and rigorous hypothesis tests for assessing variable importance. Numerical experiments demonstrate that our algorithms perform well, interpolate between regularized boosting and random forests, and confirm the validity of their built-in statistical inference procedures.