# The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies

arXiv: https://arxiv.org/abs/2007.15911 · HTML: https://mlsanity.com/paper/2007.15911/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jul 2020.

## Built on by (most cited first, 30 of 92)

- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunities](https://mlsanity.com/paper/2111.06420/) (2021)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions](https://mlsanity.com/paper/2404.03264/) (2024)
- [Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future](https://mlsanity.com/paper/2105.13137/) (2021)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [Determinants of LLM-assisted Decision-Making](https://mlsanity.com/paper/2402.17385/) (2024)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [A Concept-Based Explainability Framework for Large Multimodal Models](https://mlsanity.com/paper/2406.08074/) (2024)
- [Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Review](https://mlsanity.com/paper/2112.12596/) (2021)
- [Making Heads or Tails: Towards Semantically Consistent Visual Counterfactuals](https://mlsanity.com/paper/2203.12892/) (2022)
- [The Role of Explainability in Assuring Safety of Machine Learning in Healthcare](https://mlsanity.com/paper/2109.00520/) (2021)
- [Explainable AI for clinical and remote health applications: a survey on tabular and time series data](https://mlsanity.com/paper/2209.06528/) (2022)
- [A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?](https://mlsanity.com/paper/2304.04780/) (2023)
- [What Makes a Good Explanation?: A Harmonized View of Properties of Explanations](https://mlsanity.com/paper/2211.05667/) (2022)
- [An Accurate and Interpretable Framework for Trustworthy Process Monitoring](https://mlsanity.com/paper/2302.10426/) (2023)
- [Contrastive Explanations That Anticipate Human Misconceptions Can Improve Human Decision-Making Skills](https://mlsanity.com/paper/2410.04253/) (2024)
- [Explainability Is in the Mind of the Beholder: Establishing the Foundations of Explainable Artificial Intelligence](https://mlsanity.com/paper/2112.14466/) (2021)
- [FairCanary: Rapid Continuous Explainable Fairness](https://mlsanity.com/paper/2106.07057/) (2021)
- [Explainable Artificial Intelligence for Medical Applications: A Review](https://mlsanity.com/paper/2412.01829/) (2024)
- [Why Should I Choose You? AutoXAI: A Framework for Selecting and Tuning eXplainable AI Solutions](https://mlsanity.com/paper/2210.02795/) (2022)
- [Are Your Explanations Reliable? Investigating the Stability of LIME in Explaining Text Classifiers by Marrying XAI and Adversarial Attack](https://mlsanity.com/paper/2305.12351/) (2023)
- [Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare](https://mlsanity.com/paper/2211.16444/) (2022)
- [Why is plausibility surprisingly problematic as an XAI criterion?](https://mlsanity.com/paper/2303.17707/) (2023)
- [Explainable Depression Symptom Detection in Social Media](https://mlsanity.com/paper/2310.13664/) (2023)
- [Developing a Fidelity Evaluation Approach for Interpretable Machine Learning](https://mlsanity.com/paper/2106.08492/) (2021)
- [INSightR-Net: Interpretable Neural Network for Regression using Similarity-based Comparisons to Prototypical Examples](https://mlsanity.com/paper/2208.00457/) (2022)
- [Navigating the Synthetic Realm: Harnessing Diffusion-based Models for Laparoscopic Text-to-Image Generation](https://mlsanity.com/paper/2312.03043/) (2023)
- [A Survey of Explainable Knowledge Tracing](https://mlsanity.com/paper/2403.07279/) (2024)
- [Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond](https://mlsanity.com/paper/2309.00064/) (2023)

## Abstract (arXiv)

Artificial intelligence (AI) has huge potential to improve the health and well-being of people, but adoption in clinical practice is still limited. Lack of transparency is identified as one of the main barriers to implementation, as clinicians should be confident the AI system can be trusted. Explainable AI has the potential to overcome this issue and can be a step towards trustworthy AI. In this paper we review the recent literature to provide guidance to researchers and practitioners on the design of explainable AI systems for the health-care domain and contribute to formalization of the field of explainable AI. We argue the reason to demand explainability determines what should be explained as this determines the relative importance of the properties of explainability (i.e. interpretability and fidelity). Based on this, we propose a framework to guide the choice between classes of explainable AI methods (explainable modelling versus post-hoc explanation; model-based, attribution-based, or example-based explanations; global and local explanations). Furthermore, we find that quantitative evaluation metrics, which are important for objective standardized evaluation, are still lacking for some properties (e.g. clarity) and types of explanations (e.g. example-based methods). We conclude that explainable modelling can contribute to trustworthy AI, but the benefits of explainability still need to be proven in practice and complementary measures might be needed to create trustworthy AI in health care (e.g. reporting data quality, performing extensive (external) validation, and regulation).