# Principles of Explanation in Human-AI Systems

arXiv: https://arxiv.org/abs/2102.04972 · HTML: https://mlsanity.com/paper/2102.04972/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 70 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 10 of 10)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Generating Visual Explanations](https://mlsanity.com/paper/1603.08507/) (2016)
- [Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI](https://mlsanity.com/paper/1902.01876/) (2019)
- [Explanation Ontology: A Model of Explanations for User-Centered AI](https://mlsanity.com/paper/2010.01479/) (2020)
- [Explaining AI as an Exploratory Process: The Peircean Abduction Model](https://mlsanity.com/paper/2009.14795/) (2020)

## Built on by (most cited first, 25 of 25)

- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda](https://mlsanity.com/paper/2404.09554/) (2024)
- [Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes](https://mlsanity.com/paper/2302.05752/) (2023)
- [(Un)reasonable Allure of Ante-hoc Interpretability for High-stakes Domains: Transparency Is Necessary but Insufficient for Comprehensibility](https://mlsanity.com/paper/2306.02312/) (2023)
- [What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks](https://mlsanity.com/paper/2403.12730/) (2024)
- [Explainable AI And Visual Reasoning: Insights From Radiology](https://mlsanity.com/paper/2304.03318/) (2023)
- [Navigating Explanatory Multiverse Through Counterfactual Path Geometry](https://mlsanity.com/paper/2306.02786/) (2023)
- [A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2307.14246/) (2023)
- [Painting the black box white: experimental findings from applying XAI to an ECG reading setting](https://mlsanity.com/paper/2210.15236/) (2022)
- [Combining Cognitive and Generative AI for Self-explanation in Interactive AI Agents](https://mlsanity.com/paper/2407.18335/) (2024)
- [ReTrace: Interactive Visualizations for Reasoning Traces of Large Reasoning Models](https://mlsanity.com/paper/2511.11187/) (2025)
- [Towards Human Cognition Level-based Experiment Design for Counterfactual Explanations (XAI)](https://mlsanity.com/paper/2211.00103/) (2022)
- [Automatic Textual Explanations of Concept Lattices](https://mlsanity.com/paper/2304.08093/) (2023)
- [Understanding XAI Through the Philosopher's Lens: A Historical Perspective](https://mlsanity.com/paper/2407.18782/) (2024)
- [Using Causal Threads to Explain Changes in a Dynamic System](https://mlsanity.com/paper/2311.11334/) (2023)
- [Prompting Datasets: Data Discovery with Conversational Agents](https://mlsanity.com/paper/2312.09947/) (2023)
- [Towards an Explanation Space to Align Humans and Explainable-AI Teamwork](https://mlsanity.com/paper/2106.01503/) (2021)
- [Five policy uses of algorithmic transparency and explainability](https://mlsanity.com/paper/2302.03080/) (2023)
- [Right, No Matter Why: AI Fact-checking and AI Authority in Health-related Inquiry Settings](https://mlsanity.com/paper/2310.14358/) (2023)
- [How to Measure Human-AI Prediction Accuracy in Explainable AI Systems](https://mlsanity.com/paper/2409.00069/) (2024)
- [Explainable AI: Definition and attributes of a good explanation for health AI](https://mlsanity.com/paper/2409.15338/) (2024)
- [Analysing Explanation-Related Interactions in Collaborative Perception-Cognition-Communication-Action](https://mlsanity.com/paper/2411.12483/) (2024)
- [An Extended Symbolic-Arithmetic Model for Teaching Double-Black Removal with Rotation in Red-Black Trees](https://mlsanity.com/paper/2504.03259/) (2025)
- [Beyond single-channel agentic benchmarking](https://mlsanity.com/paper/2602.18456/) (2026)
- [Now You See Me: Designing Responsible AI Dashboards for Early-Stage Health Innovation](https://mlsanity.com/paper/2602.23378/) (2026)

## Abstract (arXiv)

Explainable Artificial Intelligence (XAI) has re-emerged in response to the development of modern AI and ML systems. These systems are complex and sometimes biased, but they nevertheless make decisions that impact our lives. XAI systems are frequently algorithm-focused; starting and ending with an algorithm that implements a basic untested idea about explainability. These systems are often not tested to determine whether the algorithm helps users accomplish any goals, and so their explainability remains unproven. We propose an alternative: to start with human-focused principles for the design, testing, and implementation of XAI systems, and implement algorithms to serve that purpose. In this paper, we review some of the basic concepts that have been used for user-centered XAI systems over the past 40 years of research. Based on these, we describe the "Self-Explanation Scorecard", which can help developers understand how they can empower users by enabling self-explanation. Finally, we present a set of empirically-grounded, user-centered design principles that may guide developers to create successful explainable systems.