# Explainability in Human-Agent Systems

arXiv: https://arxiv.org/abs/1904.08123 · HTML: https://mlsanity.com/paper/1904.08123/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2019.

## Built on by (most cited first, 30 of 34)

- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Survey](https://mlsanity.com/paper/2307.04370/) (2023)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Argumentative XAI: A Survey](https://mlsanity.com/paper/2105.11266/) (2021)
- [TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems](https://mlsanity.com/paper/2506.04133/) (2025)
- [The Who in XAI: How AI Background Shapes Perceptions of AI Explanations](https://mlsanity.com/paper/2107.13509/) (2021)
- [Question-Driven Design Process for Explainable AI User Experiences](https://mlsanity.com/paper/2104.03483/) (2021)
- [Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectives](https://mlsanity.com/paper/2407.14962/) (2024)
- [Explainable Intrusion Detection Systems (X-IDS): A Survey of Current Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2207.06236/) (2022)
- [A Critical Survey on Fairness Benefits of Explainable AI](https://mlsanity.com/paper/2310.13007/) (2023)
- [Explainable Autonomous Robots: A Survey and Perspective](https://mlsanity.com/paper/2105.02658/) (2021)
- [Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence](https://mlsanity.com/paper/2112.14466/) (2021)
- [Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond](https://mlsanity.com/paper/2309.00064/) (2023)
- [What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks](https://mlsanity.com/paper/2403.12730/) (2024)
- [A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)](https://mlsanity.com/paper/2307.14246/) (2023)
- [Justifying Social-Choice Mechanism Outcome for Improving Participant Satisfaction](https://mlsanity.com/paper/2205.15863/) (2022)
- [A Hybrid Approach for an Interpretable and Explainable Intrusion Detection System](https://mlsanity.com/paper/2111.10280/) (2021)
- [Multi-stakeholder Perspective on Responsible Artificial Intelligence and Acceptability in Education](https://mlsanity.com/paper/2402.15027/) (2024)
- [Enhancing Trust in Autonomous Agents: An Architecture for Accountability and Explainability through Blockchain and Large Language Models](https://mlsanity.com/paper/2403.09567/) (2024)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [Inductive Learning of Robot Task Knowledge from Raw Data and Online Expert Feedback](https://mlsanity.com/paper/2501.07507/) (2025)
- [Not All Explanations are Created Equal: Investigating the Pitfalls of Current XAI Evaluation](https://mlsanity.com/paper/2511.03730/) (2025)
- [An Urban Population Health Observatory for Disease Causal Pathway Analysis and Decision Support: Underlying Explainable Artificial Intelligence Model](https://mlsanity.com/paper/2208.04144/) (2022)
- [Interpretable Explainability in Facial Emotion Recognition and Gamification for Data Collection](https://mlsanity.com/paper/2211.04769/) (2022)
- [Structure and Reduction of MCTS for Explainable-AI](https://mlsanity.com/paper/2408.05488/) (2024)
- [Semifactual Explanations for Reinforcement Learning](https://mlsanity.com/paper/2409.05435/) (2024)
- [Towards LLM-generated explanations for Component-based Knowledge Graph Question Answering Systems](https://mlsanity.com/paper/2508.14553/) (2025)
- [Local Explanations for Clinical Search Engine results](https://mlsanity.com/paper/2110.12891/) (2021)
- [Introspection-based Explainable Reinforcement Learning in Episodic and Non-episodic Scenarios](https://mlsanity.com/paper/2211.12930/) (2022)

## Abstract (arXiv)

This paper presents a taxonomy of explainability in Human-Agent Systems. We consider fundamental questions about the Why, Who, What, When and How of explainability. First, we define explainability, and its relationship to the related terms of interpretability, transparency, explicitness, and faithfulness. These definitions allow us to answer why explainability is needed in the system, whom it is geared to and what explanations can be generated to meet this need. We then consider when the user should be presented with this information. Last, we consider how objective and subjective measures can be used to evaluate the entire system. This last question is the most encompassing as it will need to evaluate all other issues regarding explainability.