# Position: Explain to Question not to Justify

arXiv: https://arxiv.org/abs/2402.13914 · HTML: https://mlsanity.com/paper/2402.13914/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 21 Feb 2024.

## Builds on (most cited first, 23 of 23)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond](https://mlsanity.com/paper/2202.06861/) (2022)
- [Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI](https://mlsanity.com/paper/2003.07258/) (2020)
- [Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities](https://mlsanity.com/paper/2111.06420/) (2021)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [On the Existence of Simpler Machine Learning Models](https://mlsanity.com/paper/1908.01755/) (2019)
- [Exploring the Whole Rashomon Set of Sparse Decision Trees](https://mlsanity.com/paper/2209.08040/) (2022)
- [explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning](https://mlsanity.com/paper/1908.00087/) (2019)
- [One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparency](https://mlsanity.com/paper/2001.09734/) (2020)
- [iNNvestigate neural networks!](https://mlsanity.com/paper/1808.04260/) (2018)
- [Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvement](https://mlsanity.com/paper/2203.08008/) (2022)
- [The Grammar of Interactive Explanatory Model Analysis](https://mlsanity.com/paper/2005.00497/) (2020)
- [Understanding the (Extra-)Ordinary: Validating Deep Model Decisions with Prototypical Concept-based Explanations](https://mlsanity.com/paper/2311.16681/) (2023)
- [What Would You Ask the Machine Learning Model? Identification of User Needs for Model Explanations Based on Human-Model Conversations](https://mlsanity.com/paper/2002.05674/) (2020)
- [A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2307.14246/) (2023)
- [Be Careful When Evaluating Explanations Regarding Ground Truth](https://mlsanity.com/paper/2311.04813/) (2023)
- [i-Algebra: Towards Interactive Interpretability of Deep Neural Networks](https://mlsanity.com/paper/2101.09301/) (2021)

## Built on by (most cited first, 20 of 20)

- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning](https://mlsanity.com/paper/2501.14959/) (2025)
- [Rethinking Visual Counterfactual Explanations Through Region Constraint](https://mlsanity.com/paper/2410.12591/) (2024)
- [GraphXAIN: Narratives to Explain Graph Neural Networks](https://mlsanity.com/paper/2411.02540/) (2024)
- [Controllable Continual Test-Time Adaptation](https://mlsanity.com/paper/2405.14602/) (2024)
- [Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML](https://mlsanity.com/paper/2507.14744/) (2025)
- [Aggregated Attributions for Explanatory Analysis of 3D Segmentation Models](https://mlsanity.com/paper/2407.16653/) (2024)
- [X-SHIELD: Regularization for eXplainable Artificial Intelligence](https://mlsanity.com/paper/2404.02611/) (2024)
- [Birds look like cars: Adversarial analysis of intrinsically interpretable deep learning](https://mlsanity.com/paper/2503.08636/) (2025)
- [Model Science: getting serious about verification, explanation and control of AI systems](https://mlsanity.com/paper/2508.20040/) (2025)
- [STOOD-X methodology: using statistical nonparametric test for OOD Detection Large-Scale datasets enhanced with explainability](https://mlsanity.com/paper/2504.02685/) (2025)
- [A three-Level Framework for LLM-Enhanced eXplainable AI: From technical explanations to natural language](https://mlsanity.com/paper/2506.05887/) (2025)
- [Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributions](https://mlsanity.com/paper/2602.02560/) (2026)
- [The Case for Model Science: Verify, Explore, Steer, Refine](https://mlsanity.com/paper/2606.01189/) (2026)
- [Predictive Multiplicity in Survival Models: A Method for Quantifying Model Uncertainty in Predictive Maintenance Applications](https://mlsanity.com/paper/2504.12156/) (2025)
- [Exploring SAIG Methods for an Objective Evaluation of XAI](https://mlsanity.com/paper/2602.08715/) (2026)
- [RAB2-DEF: Dynamic and explainable defense against adversarial attacks in Federated Learning to fair poor clients](https://mlsanity.com/paper/2410.08244/) (2024)
- [NeuroFaith: Evaluating LLM Self-Explanation Faithfulness via Internal Representation Alignment](https://mlsanity.com/paper/2506.09277/) (2025)
- [Analyzing the Temporal Factors for Anxiety and Depression Symptoms with the Rashomon Perspective](https://mlsanity.com/paper/2601.20874/) (2026)
- [Selecting The Most Informative Tokens in Natural Language Autoencoders](https://mlsanity.com/paper/2609.37040/) (2026)

## Abstract (arXiv)

Explainable Artificial Intelligence (XAI) is a young but very promising field of research. Unfortunately, the progress in this field is currently slowed down by divergent and incompatible goals. We separate various threads tangled within the area of XAI into two complementary cultures of human/value-oriented explanations (BLUE XAI) and model/validation-oriented explanations (RED XAI). This position paper argues that the area of RED XAI is currently under-explored, i.e., more methods for explainability are desperately needed to question models (e.g., extract knowledge from well-performing models as well as spotting and fixing bugs in faulty models), and the area of RED XAI hides great opportunities and potential for important research necessary to ensure the safety of AI systems. We conclude this paper by presenting promising challenges in this area.