# Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection

arXiv: https://arxiv.org/abs/2109.05468 · HTML: https://mlsanity.com/paper/2109.05468/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 41 references that are in the map.
- First posted on arXiv 12 Sep 2021.

## Builds on (most cited first, 8 of 8)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Correlation and variable importance in random forests](https://mlsanity.com/paper/1310.5726/) (2013)
- [Cross-Validated Variable Selection in Tree-Based Methods Improves Predictive Performance](https://mlsanity.com/paper/1512.03444/) (2015)

## Built on by (most cited first, 6 of 6)

- [Generative AI voting: fair collective choice is resilient to LLM biases and inconsistencies](https://mlsanity.com/paper/2406.11871/) (2024)
- [Individualized and Global Feature Attributions for Gradient Boosted Trees in the Presence of ℓ2 Regularization](https://mlsanity.com/paper/2211.04409/) (2022)
- [Deep incremental learning models for financial temporal tabular datasets with distribution shifts](https://mlsanity.com/paper/2303.07925/) (2023)
- [Sequential Permutation Testing of Random Forest Variable Importance Measures](https://mlsanity.com/paper/2206.01284/) (2022)
- [Integrating Feature Selection and Machine Learning for Nitrogen Assessment in Grapevine Leaves using In-Field Hyperspectral Imaging](https://mlsanity.com/paper/2507.17869/) (2025)
- [CAFE-GB: Scalable and Stable Feature Selection for Malware Detection via Chunk-wise Aggregated Gradient Boosting](https://mlsanity.com/paper/2601.15754/) (2026)

## Abstract (arXiv)

Gradient Boosting Machines (GBM) are among the go-to algorithms on tabular data, which produce state of the art results in many prediction tasks. Despite its popularity, the GBM framework suffers from a fundamental flaw in its base learners. Specifically, most implementations utilize decision trees that are typically biased towards categorical variables with large cardinalities. The effect of this bias was extensively studied over the years, mostly in terms of predictive performance. In this work, we extend the scope and study the effect of biased base learners on GBM feature importance (FI) measures. We show that although these implementation demonstrate highly competitive predictive performance, they still, surprisingly, suffer from bias in FI. By utilizing cross-validated (CV) unbiased base learners, we fix this flaw at a relatively low computational cost. We demonstrate the suggested framework in a variety of synthetic and real-world setups, showing a significant improvement in all GBM FI measures while maintaining relatively the same level of prediction accuracy.