mlsanityData to 7 Oct 2026

A Survey on Neural Network Interpretability

arXiv 2012.14261 ↗First posted 28 Dec 2020cs.LG cs.AIcited by 291

Not on the map. Why?

Abstract (arXiv)

Along with the great success of deep neural networks, there is also growing concern about their black-box nature. The interpretability issue affects people's trust on deep learning systems. It is also related to many ethical problems, e.g., algorithmic discrimination. Moreover, interpretability is a desired property for deep networks to become powerful tools in other research fields, e.g., drug discovery and genomics. In this survey, we conduct a comprehensive review of the neural network interpretability research. We first clarify the definition of interpretability as it has been used in many different contexts. Then we elaborate on the importance of interpretability and propose a novel taxonomy organized along three dimensions: type of engagement (passive vs. active interpretation approaches), the type of explanation, and the focus (from local to global interpretability). This taxonomy provides a meaningful 3D view of distribution of papers from the relevant literature as two of the dimensions are not simply categorical but allow ordinal subcategories. Finally, we summarize the existing interpretability evaluation methods and suggest possible research directions inspired by our new taxonomy.

Key facts. Cited by 291 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 37; 2023: 63; 2024: 78; 2025: 55; 2026: 31.

291 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  2. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  3. 2023StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Modelscited by 154
  4. 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
  5. 2024Wav-KAN: Wavelet Kolmogorov-Arnold Networkscited by 118
  6. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  7. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  8. 2023A Comprehensive Survey on Deep Graph Representation Learningcited by 72
  9. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  10. 2021Synthetic Benchmarks for Scientific Research in Explainable Machine Learningcited by 59
  11. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  12. 2023Nonlinear Independent Component Analysis for Principled Disentanglement in Unsupervised Deep Learningcited by 48
38 more
  1. 2024Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Directioncited by 46
  2. 2023BayeSeg: Bayesian Modeling for Medical Image Segmentation with Interpretable Generalizabilitycited by 42
  3. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Modelscited by 37
  4. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 37
  5. 2022Guided Co-Modulated GAN for 360deg Field of View Extrapolationcited by 32
  6. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
  7. 2024TimeX++: Learning Time-Series Explanations with Information Bottleneckcited by 32
  8. 2021Unsupervised Monocular Depth Estimation in Highly Complex Environmentscited by 28
  9. 2022SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretabilitycited by 28
  10. 2023Bridging Information-Theoretic and Geometric Compression in Language Modelscited by 26
  11. 2022Human Interpretation of Saliency-based Explanation Over Textcited by 25
  12. 2024Finding Visual Task Vectorscited by 24
  13. 2023Evaluating General-Purpose AI with Psychometricscited by 23
  14. 2024Local vs. Global Interpretability: A Computational Complexity Perspectivecited by 23
  15. 2022What Makes a Good Explanation?: A Harmonized View of Properties of Explanationscited by 21
  16. 2023Neurosymbolic AI for Reasoning over Knowledge Graphs: A Surveycited by 21
  17. 2021Discovering Distinctive "Semantics" in Super-Resolution Networkscited by 20
  18. 2024Genomic Language Models: Opportunities and Challengescited by 20
  19. 2021SPECTRA: Sparse Structured Text Rationalizationcited by 19
  20. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  21. 2023Understanding and Controlling a Maze-Solving Policy Networkcited by 19
  22. 2021A Survey on Data-driven Software Vulnerability Assessment and Prioritizationcited by 18
  23. 2025A Comprehensive Survey on Self-Interpretable Neural Networkscited by 18
  24. 2024CausalFormer: An Interpretable Transformer for Temporal Causal Discoverycited by 17
  25. 2021SurvNAM: The machine learning survival model explanationcited by 16
  26. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Modelscited by 16
  27. 2022Diagnosing AI Explanation Methods with Folk Concepts of Behaviorcited by 15
  28. 2021On Recurrent Neural Networks for learning-based control: recent results and ideas for future developmentscited by 14
  29. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  30. 2024WWW: A Unified Framework for Explaining What, Where and Why of Neural Networks by Interpretation of Neuron Conceptscited by 14
  31. 2024Position: A Call to Action for a Human-Centered AutoML Paradigmcited by 14
  32. 2023A Reliable and Interpretable Framework of Multi-view Learning for Liver Fibrosis Stagingcited by 13
  33. 2023On the Interpretability of Quantum Neural Networkscited by 13
  34. 2023Interpretability of Machine Learning: Recent Advances and Future Prospectscited by 12
  35. 2024Endowing Interpretability for Neural Cognitive Diagnosis by Efficient Kolmogorov-Arnold Networkscited by 12
  36. 2024Explainable AI (XAI) in Image Segmentation in Medicine, Industry, and Beyond: A Surveycited by 11
  37. 2021When and How to Fool Explainable Models (and Humans) with Adversarial Examplescited by 10
  38. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 10

50 most cited of 291 All 291 (JSON)