# Individualized and Global Feature Attributions for Gradient Boosted Trees in the Presence of ℓ2 Regularization

arXiv: https://arxiv.org/abs/2211.04409 · HTML: https://mlsanity.com/paper/2211.04409/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 18 of 18)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Estimation and Inference of Heterogeneous Treatment Effects using Random Forests](https://mlsanity.com/paper/1510.04342/) (2015)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [Consistent Individualized Feature Attribution for Tree Ensembles](https://mlsanity.com/paper/1802.03888/) (2018)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [Feature relevance quantification in explainable AI: A causal problem](https://mlsanity.com/paper/1910.13413/) (2019)
- [Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance](https://mlsanity.com/paper/1905.03151/) (2019)
- [The age of secrecy and unfairness in recidivism prediction](https://mlsanity.com/paper/1811.00731/) (2018)
- [Iterative Random Forests to detect predictive and stable high-order interactions](https://mlsanity.com/paper/1706.08457/) (2017)
- [Unbiased Measurement of Feature Importance in Tree-Based Methods](https://mlsanity.com/paper/1903.05179/) (2019)
- [A Debiased MDI Feature Importance Measure for Random Forests](https://mlsanity.com/paper/1906.10845/) (2019)
- [Unbiased variable importance for random forests](https://mlsanity.com/paper/2003.02106/) (2020)
- [Nonparametric Feature Impact and Importance](https://mlsanity.com/paper/2006.04750/) (2020)
- [Feature Importance in Gradient Boosting Trees with Cross-Validation Feature Selection](https://mlsanity.com/paper/2109.05468/) (2021)
- [From unbiased MDI Feature Importance to Explainable AI for Trees](https://mlsanity.com/paper/2003.12043/) (2020)
- [EU General Data Protection Regulation: A Gentle Introduction](https://mlsanity.com/paper/1806.03253/) (2018)

## Built on by (most cited first, 2 of 2)

- [Interpretable Ensembles of Hyper-Rectangles as Base Models](https://mlsanity.com/paper/2303.08625/) (2023)
- [CIMLA: Interpretable AI for inference of differential causal networks](https://mlsanity.com/paper/2304.12523/) (2023)

## Abstract (arXiv)

While ℓ2 regularization is widely used in training gradient boosted trees, popular individualized feature attribution methods for trees such as Saabas and TreeSHAP overlook the training procedure. We propose Prediction Decomposition Attribution (PreDecomp), a novel individualized feature attribution for gradient boosted trees when they are trained with ℓ2 regularization. Theoretical analysis shows that the inner product between PreDecomp and labels on in-sample data is essentially the total gain of a tree, and that it can faithfully recover additive models in the population case when features are independent. Inspired by the connection between PreDecomp and total gain, we also propose TreeInner, a family of debiased global feature attributions defined in terms of the inner product between any individualized feature attribution and labels on out-sample data for each tree. Numerical experiments on a simulated dataset and a genomic ChIP dataset show that TreeInner has state-of-the-art feature selection performance. Code reproducing experiments is available at https://github.com/nalzok/TreeInner .