# Explainable AI, but explainable to whom?

arXiv: https://arxiv.org/abs/2106.05568 · HTML: https://mlsanity.com/paper/2106.05568/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2021.

## Built on by (most cited first, 7 of 7)

- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [Explainable Artificial Intelligence (XAI) from a user perspective- A synthesis of prior literature and problematizing avenues for future research](https://mlsanity.com/paper/2211.15343/) (2022)
- [Exploring How Anomalous Model Input and Output Alerts Affect Decision-Making in Healthcare](https://mlsanity.com/paper/2204.13194/) (2022)
- [Explainable AI: Definition and attributes of a good explanation for health AI](https://mlsanity.com/paper/2409.15338/) (2024)
- [Choose Your Explanation: A Comparison of SHAP and GradCAM in Human Activity Recognition](https://mlsanity.com/paper/2412.16003/) (2024)
- [Beyond Technocratic XAI: The Who, What & How in Explanation Design](https://mlsanity.com/paper/2508.09231/) (2025)
- [Explainable Machine Learning in Healthcare: Methods, Interpretation, and Applications for Clinical Research](https://mlsanity.com/paper/2608.07522/) (2026)

## Abstract (arXiv)

Advances in AI technologies have resulted in superior levels of AI-based model performance. However, this has also led to a greater degree of model complexity, resulting in 'black box' models. In response to the AI black box problem, the field of explainable AI (xAI) has emerged with the aim of providing explanations catered to human understanding, trust, and transparency. Yet, we still have a limited understanding of how xAI addresses the need for explainable AI in the context of healthcare. Our research explores the differing explanation needs amongst stakeholders during the development of an AI-system for classifying COVID-19 patients for the ICU. We demonstrate that there is a constellation of stakeholders who have different explanation needs, not just the 'user'. Further, the findings demonstrate how the need for xAI emerges through concerns associated with specific stakeholder groups i.e., the development team, subject matter experts, decision makers, and the audience. Our findings contribute to the expansion of xAI by highlighting that different stakeholders have different explanation needs. From a practical perspective, the study provides insights on how AI systems can be adjusted to support different stakeholders needs, ensuring better implementation and operation in a healthcare context.