# Learning how to explain neural networks: PatternNet and PatternAttribution

arXiv: https://arxiv.org/abs/1705.05598 · HTML: https://mlsanity.com/paper/1705.05598/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2017.

## Built on by (most cited first, 30 of 84)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation](https://mlsanity.com/paper/2206.03208/) (2022)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- [XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanations](https://mlsanity.com/paper/2103.10663/) (2021)
- [Improving Deep Learning Interpretability by Saliency Guided Training](https://mlsanity.com/paper/2111.14338/) (2021)
- [Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Off](https://mlsanity.com/paper/2209.09056/) (2022)
- [Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy](https://mlsanity.com/paper/2106.13200/) (2021)
- [Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals](https://mlsanity.com/paper/2310.00603/) (2023)
- [This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagation](https://mlsanity.com/paper/2108.12204/) (2021)
- [Explainable Deep Learning Methods in Medical Image Classification: A Survey](https://mlsanity.com/paper/2205.04766/) (2022)
- [B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers](https://mlsanity.com/paper/2306.10898/) (2023)
- [EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case](https://mlsanity.com/paper/2104.11914/) (2021)
- [Diffeomorphic Counterfactuals with Generative Models](https://mlsanity.com/paper/2206.05075/) (2022)
- [Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geoscience](https://mlsanity.com/paper/2202.03407/) (2022)
- [Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergence](https://mlsanity.com/paper/2202.03482/) (2022)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification](https://mlsanity.com/paper/2208.08741/) (2022)
- [Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset](https://mlsanity.com/paper/2103.10005/) (2021)
- [Explaining Bayesian Neural Networks](https://mlsanity.com/paper/2108.10346/) (2021)
- [Scrutinizing XAI using linear ground-truth data with suppressor variables](https://mlsanity.com/paper/2111.07473/) (2021)
- [A Game-Theoretic Taxonomy of Visual Concepts in DNNs](https://mlsanity.com/paper/2106.10938/) (2021)
- [Keep CALM and Improve Visual Feature Attribution](https://mlsanity.com/paper/2106.07861/) (2021)
- [Towards Interpretable Deep Networks for Monocular Depth Estimation](https://mlsanity.com/paper/2108.05312/) (2021)
- [Interpreting Representation Quality of DNNs for 3D Point Cloud Processing](https://mlsanity.com/paper/2111.03549/) (2021)
- [Theoretical Behavior of XAI Methods in the Presence of Suppressor Variables](https://mlsanity.com/paper/2306.01464/) (2023)
- [Saliency Cards: A Framework to Characterize and Compare Saliency Methods](https://mlsanity.com/paper/2206.02958/) (2022)
- [Explainable AI for Bioinformatics: Methods, Tools, and Applications](https://mlsanity.com/paper/2212.13261/) (2022)
- [SoK: Machine Learning Governance](https://mlsanity.com/paper/2109.10870/) (2021)

## Abstract (arXiv)

DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple neural networks. We argue that explanation methods for neural nets should work reliably in the limit of simplicity, the linear models. Based on our analysis of linear models we propose a generalization that yields two explanation techniques (PatternNet and PatternAttribution) that are theoretically sound for linear models and produce improved explanations for deep networks.