# On the Design and Evaluation of Human-centered Explainable AI Systems: A Systematic Review and Taxonomy arXiv: https://arxiv.org/abs/2510.12201 ยท HTML: https://mlsanity.com/paper/2510.12201/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 41 of its 141 references that are in the map. - First posted on arXiv 14 Oct 2025. ## Builds on (most cited first, 30 of 41) - ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016) - [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019) - [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017) - [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017) - [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018) - [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023) - [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022) - [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018) - [Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions](https://mlsanity.com/paper/1901.03729/) (2019) - [AI in Finance: Challenges, Techniques and Opportunities](https://mlsanity.com/paper/2107.09051/) (2021) - [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023) - [explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning](https://mlsanity.com/paper/1908.00087/) (2019) - [Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making](https://mlsanity.com/paper/2301.05809/) (2023) - [How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations](https://mlsanity.com/paper/2101.08758/) (2021) - [Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems](https://mlsanity.com/paper/1806.07552/) (2018) - [RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions](https://mlsanity.com/paper/2302.09466/) (2023) - [Statistical stability indices for LIME: obtaining reliable explanations for Machine Learning models](https://mlsanity.com/paper/2001.11757/) (2020) - [Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment](https://mlsanity.com/paper/1901.07694/) (2019) - [Fairness and Explainability: Bridging the Gap Towards Fair Model Explanations](https://mlsanity.com/paper/2212.03840/) (2022) - [Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorations](https://mlsanity.com/paper/2302.10671/) (2023) - [XAIR: A Framework of Explainable AI in Augmented Reality](https://mlsanity.com/paper/2303.16292/) (2023) - [Contrastive Explanations of Plans Through Model Restrictions](https://mlsanity.com/paper/2103.15575/) (2021) - [Why? Why not? When? Visual Explanations of Agent Behavior in Reinforcement Learning](https://mlsanity.com/paper/2104.02818/) (2021) - [Can Explainable AI Explain Unfairness? A Framework for Evaluating Explainable AI](https://mlsanity.com/paper/2106.07483/) (2021) - [XAI Handbook: Towards a Unified Framework for Explainable AI](https://mlsanity.com/paper/2105.06677/) (2021) - [Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation](https://mlsanity.com/paper/2109.10149/) (2021) - [Debiased-CAM to mitigate image perturbations with faithful visual explanations of machine learning](https://mlsanity.com/paper/2012.05567/) (2020) - [XAutoML: A Visual Analytics Tool for Understanding and Validating Automated Machine Learning](https://mlsanity.com/paper/2202.11954/) (2022) - [Culture-Based Explainable Human-Agent Deconfliction](https://mlsanity.com/paper/1911.10098/) (2019) - [Explainable AI and Adoption of Financial Algorithmic Advisors: an Experimental Study](https://mlsanity.com/paper/2101.02555/) (2021) ## Built on by (most cited first, 3 of 3) - [Making AI Evaluation Deployment Relevant Through Context Specification](https://mlsanity.com/paper/2603.06811/) (2026) - [Evaluation Cards for XAI Metrics](https://mlsanity.com/paper/2605.04410/) (2026) - [GFGE: Unifying Explainable AI Methods through an Interpretation Framework](https://mlsanity.com/paper/2610.05225/) (2026) ## Abstract (arXiv) As AI becomes more common in everyday living, there is an increasing demand for intelligent systems that are both performant and understandable. Explainable AI (XAI) systems aim to provide comprehensible explanations of decisions and predictions. At present, however, evaluation processes are rather technical and not sufficiently focused on the needs of human users. Consequently, evaluation studies involving human users can serve as a valuable guide for conducting user studies. This paper presents a comprehensive review of 65 user studies evaluating XAI systems across different domains and application contexts. As a guideline for XAI developers, we provide a holistic overview of the properties of XAI systems and evaluation metrics focused on human users (human-centered). We propose objectives for the human-centered design (design goals) of XAI systems. To incorporate users' specific characteristics, design goals are adapted to users with different levels of AI expertise (AI novices and data experts). In this regard, we provide an extension to existing XAI evaluation and design frameworks. The first part of our results includes the analysis of XAI system characteristics. An important finding is the distinction between the core system and the XAI explanation, which together form the whole system. Further results include the distinction of evaluation metrics into affection towards the system, cognition, usability, interpretability, and explanation metrics. Furthermore, the users, along with their specific characteristics and behavior, can be assessed. For AI novices, the relevant extended design goals include responsible use, acceptance, and usability. For data experts, the focus is performance-oriented and includes human-AI collaboration and system and user task performance.