# Model-agnostic interpretation by visualization of feature perturbations

arXiv: https://arxiv.org/abs/2101.10502 · HTML: https://mlsanity.com/paper/2101.10502/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 54 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [CatBoost: gradient boosting with categorical features support](https://mlsanity.com/paper/1810.11363/) (2018)
- [Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs](https://mlsanity.com/paper/2004.11440/) (2020)
- [ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models](https://mlsanity.com/paper/1704.01942/) (2017)
- [Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models](https://mlsanity.com/paper/1808.00196/) (2018)
- [Direct-Manipulation Visualization of Deep Networks](https://mlsanity.com/paper/1708.03788/) (2017)
- [ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion](https://mlsanity.com/paper/1910.00969/) (2019)
- [Towards Explainable Deep Learning for Credit Lending: A Case Study](https://mlsanity.com/paper/1811.06471/) (2018)
- [The Effect of Heterogeneous Data for Alzheimer's Disease Detection from Speech](https://mlsanity.com/paper/1811.12254/) (2018)
- [Automatically Explaining Machine Learning Prediction Results: A Demonstration on Type 2 Diabetes Risk Prediction](https://mlsanity.com/paper/1812.02852/) (2018)

## Abstract (arXiv)

Interpretation of machine learning models has become one of the most important research topics due to the necessity of maintaining control and avoiding bias in these algorithms. Since many machine learning algorithms are published every day, there is a need for novel model-agnostic interpretation approaches that could be used to interpret a great variety of algorithms. Thus, one advantageous way to interpret machine learning models is to feed different input data to understand the changes in the prediction. Using such an approach, practitioners can define relations among data patterns and a model's decision. This work proposes a model-agnostic interpretation approach that uses visualization of feature perturbations induced by the PSO algorithm. We validate our approach on publicly available datasets, showing the capability to enhance the interpretation of different classifiers while yielding very stable results compared with state-of-the-art algorithms.