mlsanityData to 14 Oct 2026

Learning how to explain neural networks: PatternNet and PatternAttribution

arXiv 1705.05598 ↗First posted 16 May 2017stat.ML cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

DeConvNet, Guided BackProp, LRP, were invented to better understand deep neural networks. We show that these methods do not produce the theoretically correct explanation for a linear model. Yet they are used on multi-layer networks with millions of parameters. This is a cause for concern since linear models are simple neural networks. We argue that explanation methods for neural nets should work reliably in the limit of simplicity, the linear models. Based on our analysis of linear models we propose a generalization that yields two explanation techniques (PatternNet and PatternAttribution) that are theoretically sound for linear models and produce improved explanations for deep networks.

Key facts. Cited by 84 papers in the map (data through 14 Oct 2026). First posted on arXiv 16 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 24; 2023: 13; 2024: 6; 2025: 6; 2026: 3.

84 papers.

Sort:

By area:

  1. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
  2. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  3. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  4. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 91
  5. 2021Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methodscited by 83
  6. 2021XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanationscited by 73
  7. 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
  8. 2022Concept Embedding Models: Beyond the Accuracy-Explainability Trade-Offcited by 61
  9. 2021Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAycited by 55
  10. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  11. 2021This looks more like that: Enhancing Self-Explaining Models by Prototypical Relevance Propagationcited by 43
  12. 2022Explainable Deep Learning Methods in Medical Image Classification: A Surveycited by 32
38 more
  1. 2023B-cos Alignment for Inherently Interpretable CNNs and Vision Transformerscited by 31
  2. 2021EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use casecited by 30
  3. 2022Diffeomorphic Counterfactuals with Generative Modelscited by 29
  4. 2022Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geosciencecited by 28
  5. 2022Navigating Neural Space: Revisiting Concept Activation Vectors to Overcome Directional Divergencecited by 25
  6. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  7. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19
  8. 2021Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Datasetcited by 18
  9. 2021Explaining Bayesian Neural Networkscited by 18
  10. 2021Scrutinizing XAI using linear ground-truth data with suppressor variablescited by 18
  11. 2021A Game-Theoretic Taxonomy of Visual Concepts in DNNscited by 16
  12. 2021Keep CALM and Improve Visual Feature Attributioncited by 15
  13. 2021Towards Interpretable Deep Networks for Monocular Depth Estimationcited by 12
  14. 2021Interpreting Representation Quality of DNNs for 3D Point Cloud Processingcited by 12
  15. 2023Theoretical Behavior of XAI Methods in the Presence of Suppressor Variablescited by 12
  16. 2022Saliency Cards: A Framework to Characterize and Compare Saliency Methodscited by 11
  17. 2022Explainable AI for Bioinformatics: Methods, Tools, and Applicationscited by 11
  18. 2021SoK: Machine Learning Governancecited by 10
  19. 2023Class-Discriminative Attention Maps for Vision Transformerscited by 10
  20. 2023AUTOLYCUS: Exploiting Explainable AI (XAI) for Model Extraction Attacks against Interpretable Modelscited by 9
  21. 2023Feature Perturbation Augmentation for Reliable Evaluation of Importance Estimators in Neural Networkscited by 9
  22. 2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
  23. 2021Visualizing the Emergence of Intermediate Visual Patterns in DNNscited by 7
  24. 2022SoK: Modeling Explainability in Security Analytics for Interpretability, Trustworthiness, and Usabilitycited by 7
  25. 2023XAI-TRIS: Non-linear image benchmarks to quantify false positive post-hoc attribution of feature importancecited by 7
  26. 2021Explainable Adversarial Attacks in Deep Neural Networks Using Activation Profilescited by 6
  27. 2021Evaluation of Interpretability for Deep Learning algorithms in EEG Emotion Recognition: A case study in Autismcited by 6
  28. 2022Deeply Explain CNN via Hierarchical Decompositioncited by 6
  29. 2023AGRAMPLIFIER: Defending Federated Learning Against Poisoning Attacks Through Local Update Amplificationcited by 5
  30. 2024Explainable AI needs formalizationcited by 5
  31. 2021Benchmarking Perturbation-based Saliency Maps for Explaining Atari Agentscited by 4
  32. 2021Does Dataset Complexity Matters for Model Explainers?cited by 4
  33. 2021Improved Feature Importance Computations for Tree Models: Shapley vs. Banzhafcited by 4
  34. 2023Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric researchcited by 4
  35. 2024Zero-shot Class Unlearning via Layer-wise Relevance Analysis and Neuronal Path Perturbationcited by 4
  36. 2025B-cos LM: Efficiently Transforming Pre-trained Language Models for Improved Explainabilitycited by 4
  37. 2021ExAD: An Ensemble Approach for Explanation-based Adversarial Detectioncited by 3
  38. 2021Mutual Information Preserving Back-propagation: Learn to Invert for Faithful Attributioncited by 3

50 most cited of 84 All 84 (JSON)