# iNNvestigate neural networks!

arXiv: https://arxiv.org/abs/1808.04260 · HTML: https://mlsanity.com/paper/1808.04260/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2018.

## Built on by (most cited first, 30 of 58)

- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [shapiq: Shapley Interactions for Machine Learning](https://mlsanity.com/paper/2410.01649/) (2024)
- [Explainable Deep Learning Methods in Medical Image Classification: A Survey](https://mlsanity.com/paper/2205.04766/) (2022)
- [Toward Explainable AI for Regression Models](https://mlsanity.com/paper/2112.11407/) (2021)
- [Finding the right XAI method -- A Guide for the Evaluation and Ranking of Explainable AI Methods in Climate Science](https://mlsanity.com/paper/2303.00652/) (2023)
- [Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces](https://mlsanity.com/paper/2212.14855/) (2022)
- [Interpretability-Driven Sample Selection Using Self Supervised Learning For Disease Classification And Segmentation](https://mlsanity.com/paper/2104.06087/) (2021)
- [Position: Explain to Question not to Justify](https://mlsanity.com/paper/2402.13914/) (2024)
- [Scrutinizing XAI using linear ground-truth data with suppressor variables](https://mlsanity.com/paper/2111.07473/) (2021)
- [Fooling Partial Dependence via Data Poisoning](https://mlsanity.com/paper/2105.12837/) (2021)
- [Sanity Simulations for Saliency Methods](https://mlsanity.com/paper/2105.06506/) (2021)
- [Model Explainability in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/2106.07410/) (2021)
- [Analysis of a Deep Learning Model for 12-Lead ECG Classification Reveals Learned Features Similar to Diagnostic Criteria](https://mlsanity.com/paper/2211.01738/) (2022)
- [Using Explainability to Inform Statistical Downscaling Based on Deep Learning Beyond Standard Validation Approaches](https://mlsanity.com/paper/2302.01771/) (2023)
- [Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering](https://mlsanity.com/paper/2201.05077/) (2022)
- [Uncertainty aware and explainable diagnosis of retinal disease](https://mlsanity.com/paper/2101.12041/) (2021)
- [Explain to Not Forget: Defending Against Catastrophic Forgetting with XAI](https://mlsanity.com/paper/2205.01929/) (2022)
- [XAI-TRIS: Non-linear image benchmarks to quantify false positive post-hoc attribution of feature importance](https://mlsanity.com/paper/2306.12816/) (2023)
- [COVID-19 detection using chest X-rays: is lung segmentation important for generalization?](https://mlsanity.com/paper/2104.06176/) (2021)
- [Evaluation of Interpretability for Deep Learning algorithms in EEG Emotion Recognition: A case study in Autism](https://mlsanity.com/paper/2111.13208/) (2021)
- [Gradient Backpropagation based Feature Attribution to Enable Explainable-AI on the Edge](https://mlsanity.com/paper/2210.10922/) (2022)
- [Predicting post-operative right ventricular failure using video-based deep learning](https://mlsanity.com/paper/2103.00364/) (2021)
- [Explainable Online Lane Change Predictions on a Digital Twin with a Layer Normalized LSTM and Layer-wise Relevance Propagation](https://mlsanity.com/paper/2204.01292/) (2022)
- [TSInterpret: A unified framework for time series interpretability](https://mlsanity.com/paper/2208.05280/) (2022)
- [Interpretable Deep Learning for the Remote Characterisation of Ambulation in Multiple Sclerosis using Smartphones](https://mlsanity.com/paper/2103.09171/) (2021)
- [LioNets: A Neural-Specific Local Interpretation Technique Exploiting Penultimate Layer Information](https://mlsanity.com/paper/2104.06057/) (2021)
- [Mutual Information Preserving Back-propagation: Learn to Invert for Faithful Attribution](https://mlsanity.com/paper/2104.06629/) (2021)
- [Challenges for cognitive decoding using deep learning methods](https://mlsanity.com/paper/2108.06896/) (2021)

## Abstract (arXiv)

In recent years, deep neural networks have revolutionized many application domains of machine learning and are key components of many critical decision or predictive processes. Therefore, it is crucial that domain specialists can understand and analyze actions and pre- dictions, even of the most complex neural network architectures. Despite these arguments neural networks are often treated as black boxes. In the attempt to alleviate this short- coming many analysis methods were proposed, yet the lack of reference implementations often makes a systematic comparison between the methods a major effort. The presented library iNNvestigate addresses this by providing a common interface and out-of-the- box implementation for many analysis methods, including the reference implementation for PatternNet and PatternAttribution as well as for LRP-methods. To demonstrate the versatility of iNNvestigate, we provide an analysis of image classifications for variety of state-of-the-art neural network architectures.