# Deselection of Base-Learners for Statistical Boosting -- with an Application to Distributional Regression

arXiv: https://arxiv.org/abs/2202.01657 · HTML: https://mlsanity.com/paper/2202.01657/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2022.

## Built on by (most cited first, 7 of 7)

- [A review of predictive uncertainty estimation with machine learning](https://mlsanity.com/paper/2209.08307/) (2022)
- [Additive Covariance Matrix Models: Modelling Regional Electricity Net-Demand in Great Britain](https://mlsanity.com/paper/2211.07451/) (2022)
- [Scalable Estimation for Structured Additive Distributional Regression](https://mlsanity.com/paper/2301.05593/) (2023)
- [Gradient Boosting for Spatial Regression Models with Autoregressive Disturbances](https://mlsanity.com/paper/2506.13682/) (2025)
- [Stagewise Boosting Distributional Regression](https://mlsanity.com/paper/2405.18288/) (2024)
- [MIBoost: A gradient boosting algorithm for variable selection after multiple imputation](https://mlsanity.com/paper/2507.21807/) (2025)
- [Gradient Boosting for Spatial Panel Models with Random and Fixed Effects](https://mlsanity.com/paper/2603.14543/) (2026)

## Abstract (arXiv)

We present a new procedure for enhanced variable selection for component-wise gradient boosting. Statistical boosting is a computational approach that emerged from machine learning, which allows to fit regression models in the presence of high-dimensional data. Furthermore, the algorithm can lead to data-driven variable selection. In practice, however, the final models typically tend to include too many variables in some situations. This occurs particularly for low-dimensional data (p<n), where we observe a slow overfitting behavior of boosting. As a result, more variables get included into the final model without altering the prediction accuracy. Many of these false positives are incorporated with a small coefficient and therefore have a small impact, but lead to a larger model. We try to overcome this issue by giving the algorithm the chance to deselect base-learners with minor importance. We analyze the impact of the new approach on variable selection and prediction performance in comparison to alternative methods including boosting with earlier stopping as well as twin boosting. We illustrate our approach with data of an ongoing cohort study for chronic kidney disease patients, where the most influential predictors for the health-related quality of life measure are selected in a distributional regression approach based on beta regression.