# Visualizing the Feature Importance for Black Box Models

arXiv: https://arxiv.org/abs/1804.06620 · HTML: https://mlsanity.com/paper/1804.06620/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2018.

## Built on by (most cited first, 30 of 55)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [A computational framework for physics-informed symbolic regression with straightforward integration of domain knowledge](https://mlsanity.com/paper/2209.06257/) (2022)
- [Beyond TreeSHAP: Efficient Computation of Any-Order Shapley Interactions for Tree Ensembles](https://mlsanity.com/paper/2401.12069/) (2024)
- [Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena](https://mlsanity.com/paper/2206.05487/) (2022)
- [Unifying local and global model explanations by functional decomposition of low dimensional structures](https://mlsanity.com/paper/2208.06151/) (2022)
- [PredDiff: Explanations and Interactions from Conditional Expectations](https://mlsanity.com/paper/2102.13519/) (2021)
- [XAIR: A Framework of Explainable AI in Augmented Reality](https://mlsanity.com/paper/2303.16292/) (2023)
- [Grouped Feature Importance and Combined Features Effect Plot](https://mlsanity.com/paper/2104.11688/) (2021)
- [Position: A Call to Action for a Human-Centered AutoML Paradigm](https://mlsanity.com/paper/2406.03348/) (2024)
- [Exact Computation of Any-Order Shapley Interactions for Graph Neural Networks](https://mlsanity.com/paper/2501.16944/) (2025)
- [Beyond the Black Box: Interpretability of LLMs in Finance](https://mlsanity.com/paper/2505.24650/) (2025)
- [Incremental Permutation Feature Importance (iPFI): Towards Online Explanations on Data Streams](https://mlsanity.com/paper/2209.01939/) (2022)
- [Visualizing Global Explanations of Point Cloud DNNs](https://mlsanity.com/paper/2203.09505/) (2022)
- [RHALE: Robust and Heterogeneity-aware Accumulated Local Effects](https://mlsanity.com/paper/2309.11193/) (2023)
- [Anomaly Attribution with Likelihood Compensation](https://mlsanity.com/paper/2208.10679/) (2022)
- [Statistically Valid Variable Importance Assessment through Conditional Permutations](https://mlsanity.com/paper/2309.07593/) (2023)
- [Individual Explanations in Machine Learning Models: A Survey for Practitioners](https://mlsanity.com/paper/2104.04144/) (2021)
- [Diffusion-based Visual Counterfactual Explanations -- Towards Systematic Quantitative Evaluation](https://mlsanity.com/paper/2308.06100/) (2023)
- [Model-agnostic variable importance for predictive uncertainty: an entropy-based approach](https://mlsanity.com/paper/2310.12842/) (2023)
- [Why do explanations fail? A typology and discussion on failures in XAI](https://mlsanity.com/paper/2405.13474/) (2024)
- [The Effect of Balancing Methods on Model Behavior in Imbalanced Classification Problems](https://mlsanity.com/paper/2307.00157/) (2023)
- [XplainLLM: A Knowledge-Augmented Dataset for Reliable Grounded Explanations in LLMs](https://mlsanity.com/paper/2311.08614/) (2023)
- [Coalitional strategies for efficient individual prediction explanation](https://mlsanity.com/paper/2104.00765/) (2021)
- [Free-text Rationale Generation under Readability Level Control](https://mlsanity.com/paper/2407.01384/) (2024)
- [iSAGE: An Incremental Version of SAGE for Online Explanation on Data Streams](https://mlsanity.com/paper/2303.01181/) (2023)
- [MLOps Spanning Whole Machine Learning Life Cycle: A Survey](https://mlsanity.com/paper/2304.07296/) (2023)
- [On the Relationship Between Interpretability and Explainability in Machine Learning](https://mlsanity.com/paper/2311.11491/) (2023)
- [Distribution Guided Active Feature Acquisition](https://mlsanity.com/paper/2410.03915/) (2024)
- [RbX: Region-based explanations of prediction models](https://mlsanity.com/paper/2210.08721/) (2022)
- [Explaining and visualizing black-box models through counterfactual paths](https://mlsanity.com/paper/2307.07764/) (2023)

## Abstract (arXiv)

In recent years, a large amount of model-agnostic methods to improve the transparency, trustability and interpretability of machine learning models have been developed. We introduce local feature importance as a local version of a recent model-agnostic global feature importance method. Based on local feature importance, we propose two visual tools: partial importance (PI) and individual conditional importance (ICI) plots which visualize how changes in a feature affect the model performance on average, as well as for individual observations. Our proposed methods are related to partial dependence (PD) and individual conditional expectation (ICE) plots, but visualize the expected (conditional) feature importance instead of the expected (conditional) prediction. Furthermore, we show that averaging ICI curves across observations yields a PI curve, and integrating the PI curve with respect to the distribution of the considered feature results in the global feature importance. Another contribution of our paper is the Shapley feature importance, which fairly distributes the overall performance of a model among the features according to the marginal contributions and which can be used to compare the feature importance across different models.