# Analyzing Error Sources in Global Feature Effect Estimation

arXiv: https://arxiv.org/abs/2603.15057 · HTML: https://mlsanity.com/paper/2603.15057/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 13 of 13)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models](https://mlsanity.com/paper/2007.04131/) (2020)
- [Using simulation studies to evaluate statistical methods](https://mlsanity.com/paper/1712.03198/) (2017)
- [Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery](https://mlsanity.com/paper/2206.10540/) (2022)
- [Explaining Hyperparameter Optimization via Partial Dependence Plots](https://mlsanity.com/paper/2111.04820/) (2021)
- [Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process](https://mlsanity.com/paper/2109.01433/) (2021)
- [REPID: Regional Effect Plots with implicit Interaction Detection](https://mlsanity.com/paper/2202.07254/) (2022)
- [Decomposing Global Feature Effects Based on Feature Interactions](https://mlsanity.com/paper/2306.00541/) (2023)
- [DALE: Differential Accumulated Local Effects for efficient and accurate global explanations](https://mlsanity.com/paper/2210.04542/) (2022)
- [RHALE: Robust and Heterogeneity-aware Accumulated Local Effects](https://mlsanity.com/paper/2309.11193/) (2023)
- [Accumulated Aggregated D-Optimal Designs for Estimating Main Effects in Black-Box Models](https://mlsanity.com/paper/2510.08465/) (2025)

## Built on by (most cited first, 1 of 1)

- [SAILS: Surrogate-based Analysis of Interactions via Local Effect Smooths](https://mlsanity.com/paper/2606.09404/) (2026)

## Abstract (arXiv)

Global feature effects such as partial dependence (PD) and accumulated local effects (ALE) plots are widely used to interpret black-box models. However, they are only estimates of true underlying effects, and their reliability depends on multiple sources of error. Despite the popularity of global feature effects, these error sources are largely unexplored. In particular, the practically relevant question of whether to use training or holdout data to estimate feature effects remains unanswered. We address this gap by providing a systematic, estimator-level analysis that disentangles sources of bias and variance for PD and ALE. To this end, we derive a mean-squared-error decomposition that separates model bias, estimation bias, model variance, and estimation variance, and analyze their dependence on model characteristics, data selection, and sample size. We validate our theoretical findings through an extensive simulation study across multiple data-generating processes, learners, estimation strategies (training data, validation data, and cross-validation), and sample sizes. Our results reveal that, while using holdout data is theoretically the cleanest, potential biases arising from the training data are empirically negligible and dominated by the impact of the usually higher sample size. The estimation variance depends on both the presence of interactions and the sample size, with ALE being particularly sensitive to the latter. Cross-validation-based estimation is a promising approach that reduces the model variance component, particularly for overfitting models. Our analysis provides a principled explanation of the sources of error in feature effect estimates and offers concrete guidance on choosing estimation strategies when interpreting machine learning models.