# Boosting Random Forests to Reduce Bias; One-Step Boosted Forest and its Variance Estimate

arXiv: https://arxiv.org/abs/1803.08000 · HTML: https://mlsanity.com/paper/1803.08000/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2018.

## Built on by (most cited first, 8 of 8)

- [Why do Random Forests Work? Understanding Tree Ensembles as Self-Regularizing Adaptive Smoothers](https://mlsanity.com/paper/2402.01502/) (2024)
- [The Infinitesimal Jackknife and Combinations of Models](https://mlsanity.com/paper/2209.00147/) (2022)
- [Generalised Boosted Forests](https://mlsanity.com/paper/2102.12561/) (2021)
- [RFpredInterval: An R Package for Prediction Intervals with Random Forests and Boosted Forests](https://mlsanity.com/paper/2106.08217/) (2021)
- [Layer-based Composite Reputation Bootstrapping](https://mlsanity.com/paper/2102.09951/) (2021)
- [A generalized decision tree ensemble based on the NeuralNetworks architecture: Distributed Gradient Boosting Forest (DGBF)](https://mlsanity.com/paper/2402.03386/) (2024)
- [Riemann-Lebesgue Forest for Regression](https://mlsanity.com/paper/2402.04550/) (2024)
- [Subsampled Ensemble Can Improve Generalization Tail Exponentially](https://mlsanity.com/paper/2405.14741/) (2024)

## Abstract (arXiv)

In this paper we propose using the principle of boosting to reduce the bias of a random forest prediction in the regression setting. From the original random forest fit we extract the residuals and then fit another random forest to these residuals. We call the sum of these two random forests a textitone-step boosted forest. We show with simulated and real data that the one-step boosted forest has a reduced bias compared to the original random forest. The paper also provides a variance estimate of the one-step boosted forest by an extension of the infinitesimal Jackknife estimator. Using this variance estimate we can construct prediction intervals for the boosted forest and we show that they have good coverage probabilities. Combining the bias reduction and the variance estimate we show that the one-step boosted forest has a significant reduction in predictive mean squared error and thus an improvement in predictive performance. When applied on datasets from the UCI database, one-step boosted forest performs better than random forest and gradient boosting machine algorithms. Theoretically we can also extend such a boosting process to more than one step and the same principles outlined in this paper can be used to find variance estimates for such predictors. Such boosting will reduce bias even further but it risks over-fitting and also increases the computational burden.