mlsanityData to 7 Oct 2026

The Mythos of Model Interpretability

arXiv 1606.03490 ↗First posted 10 Jun 2016cs.LG cs.AI cs.CVcited by 716

Not on the map. Why?

Abstract (arXiv)

Supervised machine learning models boast remarkable predictive capabilities. But can you trust your model? Will it work in deployment? What else can it tell you about the world? We want models to be not only good, but interpretable. And yet the task of interpretation appears underspecified. Papers provide diverse and sometimes non-overlapping motivations for interpretability, and offer myriad notions of what attributes render models interpretable. Despite this ambiguity, many papers proclaim interpretability axiomatically, absent further explanation. In this paper, we seek to refine the discourse on interpretability. First, we examine the motivations underlying interest in interpretability, finding them to be diverse and occasionally discordant. Then, we address model properties and techniques thought to confer interpretability, identifying transparency to humans and post-hoc explanations as competing notions. Throughout, we discuss the feasibility and desirability of different notions, and question the oft-made assertions that linear models are interpretable and that deep neural networks are not.

Key facts. Cited by 716 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 2; 2020: 1; 2021: 190; 2022: 126; 2023: 120; 2024: 100; 2025: 94; 2026: 83.

716 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  2. 2021A Survey on Automated Fact-Checkingcited by 376
  3. 2023Large Language Model Alignment: A Surveycited by 241
  4. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  5. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  6. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  7. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  8. 2021FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured informationcited by 166
  9. 2022A Review on Language Models as Knowledge Basescited by 156
  10. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  11. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  12. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
38 more
  1. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  2. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  3. 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
  4. 2022OpenXAI: Towards a Transparent Evaluation of Model Explanationscited by 125
  5. 2021Expanding Explainability: Towards Social Transparency in AI systemscited by 124
  6. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  7. 2021What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Researchcited by 119
  8. 2021Interpretable Deep Learning: Interpretation, Interpretability, Trustworthiness, and Beyondcited by 116
  9. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  10. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  11. 2021A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Conceptscited by 90
  12. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 85
  13. 2021Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methodscited by 83
  14. 2021If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniquescited by 82
  15. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  16. 2021Explainability of deep vision-based autonomous driving systems: Review and challengescited by 80
  17. 2021Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Casescited by 77
  18. 2023Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblockscited by 77
  19. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  20. 2023Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanationscited by 73
  21. 2021Explaining Time Series Predictions with Dynamic Maskscited by 72
  22. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  23. 2021Improving Deep Learning Interpretability by Saliency Guided Trainingcited by 68
  24. 2021Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needscited by 64
  25. 2023Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulationcited by 60
  26. 2021Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretationscited by 59
  27. 2021Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applicationscited by 59
  28. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  29. 2021Understanding the Effect of Out-of-distribution Examples and Interactive Explanations on Human-AI Decision Makingcited by 58
  30. 2021How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanationscited by 58
  31. 2022Concept Activation Regions: A Generalized Framework For Concept-Based Explanationscited by 57
  32. 2021Socially Responsible AI Algorithms: Issues, Purposes, and Challengescited by 56
  33. 2021Incorporating Domain Knowledge into Deep Neural Networkscited by 56
  34. 2022Framework for Evaluating Faithfulness of Local Explanationscited by 55
  35. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  36. 2023On Early Detection of Hallucinations in Factual Question Answeringcited by 54
  37. 2021Explanation-Based Human Debugging of NLP Models: A Surveycited by 53
  38. 2022Evaluation of post-hoc interpretability methods in time-series classificationcited by 53

50 most cited of 716 All 716 (JSON)