# Incremental Permutation Feature Importance (iPFI): Towards Online Explanations on Data Streams

arXiv: https://arxiv.org/abs/2209.01939 · HTML: https://mlsanity.com/paper/2209.01939/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 66 references that are in the map.
- First posted on arXiv 5 Sep 2022.

## Builds on (most cited first, 23 of 23)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Explaining by Removing: A Unified Framework for Model Explanation](https://mlsanity.com/paper/2011.14878/) (2020)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [Feature relevance quantification in explainable AI: A causal problem](https://mlsanity.com/paper/1910.13413/) (2019)
- [Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regression](https://mlsanity.com/paper/2012.01536/) (2020)
- [True to the Model or True to the Data?](https://mlsanity.com/paper/2006.16234/) (2020)
- [FastSHAP: Real-Time Shapley Value Estimation](https://mlsanity.com/paper/2107.07436/) (2021)
- [River: machine learning for streaming data in Python](https://mlsanity.com/paper/2012.04740/) (2020)
- [Shapley explainability on the data manifold](https://mlsanity.com/paper/2006.01272/) (2020)
- [Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance](https://mlsanity.com/paper/1905.03151/) (2019)
- [OpenML-Python: an extensible Python API for OpenML](https://mlsanity.com/paper/1911.02490/) (2019)
- [Challenges in Benchmarking Stream Learning Algorithms with Real-world Data](https://mlsanity.com/paper/2005.00113/) (2020)
- [Visualizing the Feature Importance for Black Box Models](https://mlsanity.com/paper/1804.06620/) (2018)
- [Correlation and variable importance in random forests](https://mlsanity.com/paper/1310.5726/) (2013)
- [Model-agnostic Feature Importance and Effects with Dependent Features -- A Conditional Subgroup Approach](https://mlsanity.com/paper/2006.04628/) (2020)
- [Grouped variable importance with random forests and application to multiple functional data analysis](https://mlsanity.com/paper/1411.4170/) (2014)
- [Relative Feature Importance](https://mlsanity.com/paper/2007.08283/) (2020)
- [Progressive Data Science: Potential and Challenges](https://mlsanity.com/paper/1812.08032/) (2018)
- [Explainable AI for tailored electricity consumption feedback -- an experimental evaluation of visualizations](https://mlsanity.com/paper/2208.11408/) (2022)

## Built on by (most cited first, 9 of 9)

- [Model Based Explanations of Concept Drift](https://mlsanity.com/paper/2303.09331/) (2023)
- [iPDP: On Partial Dependence Plots in Dynamic Modeling Scenarios](https://mlsanity.com/paper/2306.07775/) (2023)
- [iSAGE: An Incremental Version of SAGE for Online Explanation on Data Streams](https://mlsanity.com/paper/2303.01181/) (2023)
- [Explainable Malware Detection with Tailored Logic Explained Networks](https://mlsanity.com/paper/2405.03009/) (2024)
- [Towards Explainable Deep Clustering for Time Series Data](https://mlsanity.com/paper/2507.20840/) (2025)
- [Hierarchical Ensemble-Based Feature Selection for Time Series Forecasting](https://mlsanity.com/paper/2310.17544/) (2023)
- [From XAI to MLOps: Explainable Concept Drift Detection with Profile Drift Detection](https://mlsanity.com/paper/2412.11308/) (2024)
- [Dynamic Features Adaptation in Networking: Toward Flexible training and Explainable inference](https://mlsanity.com/paper/2510.08303/) (2025)
- [FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems](https://mlsanity.com/paper/2610.13806/) (2026)

## Abstract (arXiv)

Explainable Artificial Intelligence (XAI) has mainly focused on static learning scenarios so far. We are interested in dynamic scenarios where data is sampled progressively, and learning is done in an incremental rather than a batch mode. We seek efficient incremental algorithms for computing feature importance (FI) measures, specifically, an incremental FI measure based on feature marginalization of absent features similar to permutation feature importance (PFI). We propose an efficient, model-agnostic algorithm called iPFI to estimate this measure incrementally and under dynamic modeling conditions including concept drift. We prove theoretical guarantees on the approximation quality in terms of expectation and variance. To validate our theoretical findings and the efficacy of our approaches compared to traditional batch PFI, we conduct multiple experimental studies on benchmark data with and without concept drift.