mlsanityData to 14 Oct 2026

Interpretability Needs a New Paradigm

arXiv 2405.05386 ↗First posted 8 May 2024cs.LG cs.CL cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

Interpretability is the study of explaining models in understandable terms to humans. At present, interpretability is divided into two paradigms: the intrinsic paradigm, which believes that only models designed to be explained can be explained, and the post-hoc paradigm, which believes that black-box models can be explained. At the core of this debate is how each paradigm ensures its explanations are faithful, i.e., true to the model's behavior. This is important, as false but convincing explanations lead to unsupported confidence in artificial intelligence (AI), which can be dangerous. This paper's position is that we should think about new paradigms while staying vigilant regarding faithfulness. First, by examining the history of paradigms in science, we see that paradigms are constantly evolving. Then, by examining the current paradigms, we can understand their underlying beliefs, the value they bring, and their limitations. Finally, this paper presents 3 emerging paradigms for interpretability. The first paradigm designs models such that faithfulness can be easily measured. Another optimizes models such that explanations become faithful. The last paradigm proposes to develop models that produce both a prediction and an explanation.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 78 of its 120 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

78 of 120 references.

References · 78

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  6. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  7. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  9. 2018Sanity Checks for Saliency Mapscited by 1,156
  10. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  12. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
38 more
  1. 2019Attention is not Explanationcited by 929
  2. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  3. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  5. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  6. 2016The Mythos of Model Interpretabilitycited by 721
  7. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  8. 2019Designing and Interpreting Probes with Control Taskscited by 640
  9. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  10. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  11. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  13. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  14. 2015Neural Module Networkscited by 376
  15. 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 344
  16. 2015Understanding Neural Networks Through Deep Visualizationcited by 334
  17. 2017The (Un)reliability of saliency methodscited by 333
  18. 2019On the (In)fidelity and Sensitivity for Explanationscited by 310
  19. 2019Interpretable machine learning: definitions, methods, and applicationscited by 305
  20. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  21. 2018Learning to Explain: An Information-Theoretic Perspective on Model Interpretationcited by 248
  22. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  23. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  24. 2016Understanding Neural Networks through Representation Erasurecited by 224
  25. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  26. 2019Explainable Machine Learning in Deploymentcited by 211
  27. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 199
  28. 2018Representer Point Selection for Explaining Deep Neural Networkscited by 187
  29. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  30. 2019One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniquescited by 164
  31. 2015Visualizing and Understanding Recurrent Networkscited by 160
  32. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  33. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  34. 2020The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?cited by 122
  35. 2020Drug discovery with explainable artificial intelligencecited by 118
  36. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  37. 2024Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Modelscited by 105
  38. 2024Are self-explanations from Large Language Models faithful?cited by 101

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 4.

11 papers.

  1. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  2. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  3. 2025Surrogate Modeling and Explainable Artificial Intelligence for Complex Systems: A Workflow for Automated Simulation Explorationcited by 6
  4. 2024DLBacktrace: A Model Agnostic Explainability for any Deep Learning Modelscited by 2
  5. 2025Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoringcited by 2
  6. 2025Tell me why: Visual foundation models as self-explainable classifierscited by 2
  7. 2024In Defence of Post-hoc Explainabilitycited by 1
  8. 2026From Mechanistic to Compositional Interpretabilitycited by 1
  9. 2026Scaling Inherently Interpretable Language Modelscited by 1
  10. 2026Analyzing Shapley Additive Explanations to Understand Anomaly Detection Algorithm Behaviors and Their Complementaritycited by 0
  11. 2026Towards Faithful Multimodal Concept Bottleneck Modelscited by 0