# Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI

arXiv: https://arxiv.org/abs/2308.06274 · HTML: https://mlsanity.com/paper/2308.06274/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 28 of 28)

- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems](https://mlsanity.com/paper/2001.08298/) (2020)
- [What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use](https://mlsanity.com/paper/1905.05134/) (2019)
- [OpenXAI: Towards a Transparent Evaluation of Model Explanations](https://mlsanity.com/paper/2206.11104/) (2022)
- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](https://mlsanity.com/paper/2110.10790/) (2021)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019)
- [The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies](https://mlsanity.com/paper/2007.15911/) (2020)
- [Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation](https://mlsanity.com/paper/2003.03934/) (2020)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs](https://mlsanity.com/paper/2101.09824/) (2021)
- [Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI](https://mlsanity.com/paper/2206.10847/) (2022)
- [A Study of Automatic Metrics for the Evaluation of Natural Language Explanations](https://mlsanity.com/paper/2103.08545/) (2021)
- [Question-Driven Design Process for Explainable AI User Experiences](https://mlsanity.com/paper/2104.03483/) (2021)
- [Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent](https://mlsanity.com/paper/2007.07422/) (2020)
- [A Survey of Explainable Graph Neural Networks: Taxonomy and Evaluation Metrics](https://mlsanity.com/paper/2207.12599/) (2022)
- [Machine Explanations and Human Understanding](https://mlsanity.com/paper/2202.04092/) (2022)
- [Counterfactual Evaluation for Explainable AI](https://mlsanity.com/paper/2109.01962/) (2021)
- [In AI We Trust? Factors That Influence Trustworthiness of AI-infused Decision-Making Processes](https://mlsanity.com/paper/1912.02675/) (2019)
- [Roadmap of Designing Cognitive Metrics for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2108.01737/) (2021)
- [Developing a Fidelity Evaluation Approach for Interpretable Machine Learning](https://mlsanity.com/paper/2106.08492/) (2021)
- [A Meta Survey of Quality Evaluation Criteria in Explanation Methods](https://mlsanity.com/paper/2203.13929/) (2022)
- [Benchmarking Instance-Centric Counterfactual Algorithms for XAI: From White Box to Black Box](https://mlsanity.com/paper/2203.02399/) (2022)
- [Qualitative Investigation in Explainable Artificial Intelligence: A Bit More Insight from Social Science](https://mlsanity.com/paper/2011.07130/) (2020)

## Built on by (most cited first, 3 of 3)

- [Interpretation of Intracardiac Electrograms Through Textual Representations](https://mlsanity.com/paper/2402.01115/) (2024)
- [See What I Mean? CUE: A Cognitive Model of Understanding Explanations](https://mlsanity.com/paper/2506.14775/) (2025)
- [Counterfactual Explanations for Time Series Should be Human-Centered and Temporally Coherent in Interventions](https://mlsanity.com/paper/2512.14559/) (2025)

## Abstract (arXiv)

While research on explainable AI (XAI) is booming and explanation techniques have proven promising in many application domains, standardised human-centred evaluation procedures are still missing. In addition, current evaluation procedures do not assess XAI methods holistically in the sense that they do not treat explanations' effects on humans as a complex user experience. To tackle this challenge, we propose to adapt the User-Centric Evaluation Framework used in recommender systems: we integrate explanation aspects, summarise explanation properties, indicate relations between them, and categorise metrics that measure these properties. With this comprehensive evaluation framework, we hope to contribute to the human-centred standardisation of XAI evaluation.