mlsanityData to 7 Oct 2026

Diagnosing AI Explanation Methods with Folk Concepts of Behavior

arXiv 2201.11239 ↗First posted 27 Jan 2022cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

We investigate a formalism for the conditions of a successful explanation of AI. We consider "success" to depend not only on what information the explanation contains, but also on what information the human explainee understands from it. Theory of mind literature discusses the folk concepts that humans use to understand and generalize behavior. We posit that folk concepts of behavior provide us with a "language" that humans understand behavior with. We use these folk concepts as a framework of social attribution by the human explainee - the information constructs that humans are likely to comprehend from explanations - by introducing a blueprint for an explanatory narrative (Figure 1) that explains AI behavior with these constructs. We then demonstrate that many XAI methods today can be mapped to folk concepts of behavior in a qualitative evaluation. This allows us to uncover their failure modes that prevent current methods from explaining successfully - i.e., the information constructs that are missing for any given XAI method, and whose inclusion can decrease the likelihood of misunderstanding AI behavior.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 74 of its 154 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

74 of 154 references.

References · 74

  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  5. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  8. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  9. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  10. 2018Sanity Checks for Saliency Mapscited by 1,149
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  12. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
38 more
  1. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  2. 2020Quantifying Attention Flow in Transformerscited by 759
  3. 2016The Mythos of Model Interpretabilitycited by 716
  4. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  5. 2019Attention is not not Explanationcited by 629
  6. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  7. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 542
  8. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  9. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 467
  10. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  11. 2017Interpretation of Neural Networks is Fragilecited by 443
  12. 2021Causal Abstractions of Neural Networkscited by 401
  13. 2017The (Un)reliability of saliency methodscited by 331
  14. 2018On the Robustness of Interpretability Methodscited by 310
  15. 2020A Survey on Neural Network Interpretabilitycited by 291
  16. 2016Rationalizing Neural Predictionscited by 281
  17. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  18. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 230
  19. 2020A Survey on the Explainability of Supervised Machine Learningcited by 225
  20. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  21. 2015Visualizing and Understanding Neural Models in NLPcited by 199
  22. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 189
  23. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  24. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  25. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  26. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  27. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  28. 2018Pathologies of Neural Models Make Interpretations Difficultcited by 140
  29. 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 138
  30. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  31. 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
  32. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  33. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  34. 2019Rethinking Cooperative Rationalization: Introspective Extraction and Complement Controlcited by 96
  35. 2020Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?cited by 95
  36. 2020Aligning Faithful Interpretations with their Social Attributioncited by 79
  37. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  38. 2021"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classificationcited by 66

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 1; 2025: 5; 2026: 1.

15 papers.

Sort:

By area:

  1. 2025Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistantcited by 26
  2. 2022Attribution-based Explanations that Provide Recourse Cannot be Robustcited by 20
  3. 2024LLMCheckup: Conversational Examination of Large Language Models via Interpretability Tools and Self-Explanationscited by 16
  4. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  5. 2022Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methodscited by 13
  6. 2023InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanationscited by 11
  7. 2022Calibrating Trust of Multi-Hop Question Answering Systems with Decompositional Probescited by 9
  8. 2022Mediators: Conversational Agents Explaining NLP Model Behaviorcited by 7
  9. 2023Benchmarking Faithfulness: Towards Accurate Natural Language Explanations in Vision-Language Taskscited by 2
  10. 2025Reward Models are Metrics in a Trench Coatcited by 2
  11. 2022Challenges in Explanation Quality Evaluationcited by 1
  12. 2025Back to the Baseline: Examining Baseline Effects on Explainability Metricscited by 1
3 more
  1. 2025The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflowscited by 0
  2. 2025Beyond Technocratic XAI: The Who, What & How in Explanation Designcited by 0
  3. 2026Conversational versus Dashboard Explainable AI for UAV Intrusion Detection: An Empirical Study of Operator Trust and Reliancecited by 0