# What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks

arXiv: https://arxiv.org/abs/2403.12730 · HTML: https://mlsanity.com/paper/2403.12730/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 85 references that are in the map.
- First posted on arXiv 19 Mar 2024.

## Builds on (most cited first, 30 of 42)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Explainable Machine Learning for Scientific Insights and Discoveries](https://mlsanity.com/paper/1905.08883/) (2019)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019)
- [Do Feature Attribution Methods Correctly Attribute Features?](https://mlsanity.com/paper/2104.14403/) (2021)
- [The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies](https://mlsanity.com/paper/2007.15911/) (2020)
- [Benchmarking Attribution Methods with Relative Feature Importance](https://mlsanity.com/paper/1907.09701/) (2019)
- [Explainable Artificial Intelligence: a Systematic Review](https://mlsanity.com/paper/2006.00093/) (2020)
- [How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1802.00682/) (2018)
- [An Evaluation of the Human-Interpretability of Explanation](https://mlsanity.com/paper/1902.00006/) (2019)
- [On quantitative aspects of model interpretability](https://mlsanity.com/paper/2007.07584/) (2020)
- [Measuring the Quality of Explanations: The System Causability Scale (SCS). Comparing Human and Machine Explanations](https://mlsanity.com/paper/1912.09024/) (2019)
- [One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparency](https://mlsanity.com/paper/2001.09734/) (2020)
- [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021)
- [The Pragmatic Turn in Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2002.09595/) (2020)
- [The Promise and Peril of Human Evaluation for Model Interpretability](https://mlsanity.com/paper/1711.07414/) (2017)
- [Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems](https://mlsanity.com/paper/1806.07552/) (2018)
- [Stakeholders in Explainable AI](https://mlsanity.com/paper/1810.00184/) (2018)
- [Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI](https://mlsanity.com/paper/2206.10847/) (2022)
- [Explainability in Human-Agent Systems](https://mlsanity.com/paper/1904.08123/) (2019)
- [Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI](https://mlsanity.com/paper/2302.00799/) (2023)
- [XFake: Explainable Fake News Detector with Visualizations](https://mlsanity.com/paper/1907.07757/) (2019)

## Built on by (most cited first, 6 of 6)

- [LLMs for Explainable AI: A Comprehensive Survey](https://mlsanity.com/paper/2504.00125/) (2025)
- [Uncertainty Quantification as a Principled Foundation for Explainable Artificial Intelligence: A Case Study of Counterfactual Explanations](https://mlsanity.com/paper/2502.17007/) (2025)
- [Position: Explainability Research Must Prioritize Foundations over Ad-hoc Methods](https://mlsanity.com/paper/2607.14123/) (2026)
- [Textual Explanations and Their Evaluations for Reinforcement Learning Policy](https://mlsanity.com/paper/2601.02514/) (2026)
- [Emergent, not Immanent: A Baradian Reading of Explainable AI](https://mlsanity.com/paper/2601.15029/) (2026)
- [XAI-Arena: Can LLMs Assess the Quality of XAI Explanations?](https://mlsanity.com/paper/2609.09428/) (2026)

## Abstract (arXiv)

Despite significant progress, evaluation of explainable artificial intelligence remains elusive and challenging. In this paper we propose a fine-grained validation framework that is not overly reliant on any one facet of these sociotechnical systems, and that recognises their inherent modular structure: technical building blocks, user-facing explanatory artefacts and social communication protocols. While we concur that user studies are invaluable in assessing the quality and effectiveness of explanation presentation and delivery strategies from the explainees' perspective in a particular deployment context, the underlying explanation generation mechanisms require a separate, predominantly algorithmic validation strategy that accounts for the technical and human-centred desiderata of their (numerical) outputs. Such a comprehensive sociotechnical utility-based evaluation framework could allow to systematically reason about the properties and downstream influence of different building blocks from which explainable artificial intelligence systems are composed -- accounting for a diverse range of their engineering and social aspects -- in view of the anticipated use case.