mlsanityData to 14 Oct 2026

Position: Explainability Research Must Prioritize Foundations over Ad-hoc Methods

arXiv 2607.14123 ↗First posted 18 Jun 2026cs.LG cs.AIcited by 2

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Despite the proliferation of Explainable AI (XAI) techniques -- from feature attributions to sparse autoencoders -- explanations rarely influence real-world workflows. In practice, they are often generated and discarded without guiding meaningful action. This gap reflects foundational shortcomings: research has not yet established methodologies for integrating explanations into end-to-end, human-in-the-loop systems. This position paper argues that the machine learning community must pivot from ad-hoc XAI methods toward addressing foundational & structural challenges, including unclear problem formulations, underspecified evaluation objectives, and the absence of pipelines for explanation-driven feedback. We support this claim through an analysis of recent ICML, NeurIPS, and ICLR papers and a survey of XAI practitioners, revealing recurring issues that limit cumulative progress. We conclude by outlining a practical checklist designed to shift XAI toward a more human-centered, action-oriented paradigm. By emphasizing foundational clarity over the development of ad-hoc methods, we hope to provide a roadmap for integrating explanations into actionable, feedback-driven AI systems.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 57 of its 81 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

57 of 81 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Open Problems in Mechanistic Interpretability27 Jan
  2. 2024Mechanistic?7 Oct

Other references · 55

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  6. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  7. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  8. 2018Sanity Checks for Saliency Mapscited by 1,156
  9. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  10. 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,089
  11. 2022Toy Models of Superpositioncited by 971
  12. 2019Attention is not Explanationcited by 929
38 more
  1. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  2. 2019Attention is not not Explanationcited by 636
  3. 2019Explaining Machine Learning Classifiers through Diverse Counterfactual Explanationscited by 614
  4. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  5. 2024Scaling and evaluating sparse autoencoderscited by 553
  6. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  7. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
  8. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  9. 2017The (Un)reliability of saliency methodscited by 333
  10. 2018Actionable Recourse in Linear Classificationcited by 328
  11. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
  12. 2020Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Surveycited by 254
  13. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  14. 2019On Completeness-aware Concept-Based Explanations in Deep Neural Networkscited by 239
  15. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  16. 2019Explainable Machine Learning in Deploymentcited by 211
  17. 2019One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniquescited by 164
  18. 2022The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspectivecited by 159
  19. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  20. 2021Human-Centered Explainable AI (XAI): From Algorithms to User Experiencescited by 123
  21. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
  22. 2021What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Researchcited by 119
  23. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  24. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  25. 2021If Only We Had Better Counterfactual Explanations: Five Key Deficits to Rectify in the Evaluation of Counterfactual XAI Techniquescited by 82
  26. 2022Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanationscited by 77
  27. 2020Human Factors in Model Interpretability: Industry Practices, Challenges, and Needscited by 74
  28. 2021IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammographycited by 72
  29. 2021Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needscited by 64
  30. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  31. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  32. 2022Framework for Evaluating Faithfulness of Local Explanationscited by 55
  33. 2022Concept-level Debugging of Part-Prototype Networkscited by 52
  34. 2022Adaptive Testing of Computer Vision Modelscited by 39
  35. 2025Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistenciescited by 39
  36. 2020What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoorscited by 32
  37. 2024Towards Unifying Interpretability and Control: Evaluation via Interventioncited by 32
  38. 2024From Explainable to Interactive AI: A Literature Review on Current Trends in Human-AI Interactioncited by 26

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Explaining Image Similarity with Automatically Extracted Concept Activation Vectorscited by 0
  2. 2026Challenges in Evaluating Explanation Methods for Static and Evolving Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Explainable artificial intelligence (XAI): from inherent explainability to large language models
  2. 2026Explainable AI: Learning from the Learners
  3. 2024Towards Explainable Artificial Intelligence (XAI): A Data Mining Perspective
  4. 2025Towards Transparent AI: A Survey on Explainable Large Language Models
  5. 2023Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions
  6. 2022From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent
  7. 2022Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning
  8. 2026Addressing the Selection Problem in Explainable AI