# A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?

arXiv: https://arxiv.org/abs/2304.04780 · HTML: https://mlsanity.com/paper/2304.04780/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 70 of its 202 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 30 of 70)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [A Benchmark for Interpretability Methods in Deep Neural Networks](https://mlsanity.com/paper/1806.10758/) (2018)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [Manipulating and Measuring Model Interpretability](https://mlsanity.com/paper/1802.07810/) (2018)
- [Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey](https://mlsanity.com/paper/2006.11371/) (2020)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model](https://mlsanity.com/paper/1511.01644/) (2015)
- [Explaining Explanations in AI](https://mlsanity.com/paper/1811.01439/) (2018)
- [What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use](https://mlsanity.com/paper/1905.05134/) (2019)
- [What do we need to build explainable AI systems for the medical domain?](https://mlsanity.com/paper/1712.09923/) (2017)

## Built on by (most cited first, 24 of 24)

- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [CoE: Chain-of-Explanation via Automatic Visual Concept Circuit Description and Polysemanticity Quantification](https://mlsanity.com/paper/2503.15234/) (2025)
- [Investigating the Duality of Interpretability and Explainability in Machine Learning](https://mlsanity.com/paper/2503.21356/) (2025)
- [StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models](https://mlsanity.com/paper/2311.16509/) (2023)
- [The Significance of Machine Learning in Clinical Disease Diagnosis: A Review](https://mlsanity.com/paper/2310.16978/) (2023)
- [QUCE: The Minimisation and Quantification of Path-Based Uncertainty for Generative Counterfactual Explanations](https://mlsanity.com/paper/2402.17516/) (2024)
- [Interpretable Machine Learning for Survival Analysis](https://mlsanity.com/paper/2403.10250/) (2024)
- [A Design Framework for operationalizing Trustworthy Artificial Intelligence in Healthcare: Requirements, Tradeoffs and Challenges for its Clinical Adoption](https://mlsanity.com/paper/2504.19179/) (2025)
- [Explaining Human Activity Recognition with SHAP: Validating Insights with Perturbation and Quantitative Measures](https://mlsanity.com/paper/2411.03714/) (2024)
- [Modeling Quantum Machine Learning for Genomic Data Analysis](https://mlsanity.com/paper/2501.08193/) (2025)
- [Visual-Conversational Interface for Evidence-Based Explanation of Diabetes Risk Prediction](https://mlsanity.com/paper/2507.02920/) (2025)
- [Explainable AI in Orthopedics: Challenges, Opportunities, and Prospects](https://mlsanity.com/paper/2308.04696/) (2023)
- [An interpretable prototype parts-based neural network for medical tabular data](https://mlsanity.com/paper/2603.05423/) (2026)
- [CARNA: Characterizing Advanced heart failure Risk and hemodyNAmic phenotypes using learned multi-valued decision diagrams](https://mlsanity.com/paper/2306.06801/) (2023)
- [Deciphering knee osteoarthritis diagnostic features with explainable artificial intelligence: A systematic review](https://mlsanity.com/paper/2308.09380/) (2023)
- [Choose Your Explanation: A Comparison of SHAP and GradCAM in Human Activity Recognition](https://mlsanity.com/paper/2412.16003/) (2024)
- [Scaling Electronic Health Record Foundation Models for Population Health Management](https://mlsanity.com/paper/2506.00209/) (2025)
- [ITL-LIME: Instance-Based Transfer Learning for Enhancing Local Explanations in Low-Resource Data Settings](https://mlsanity.com/paper/2508.13672/) (2025)
- [Governable AI: Provable Safety Under Extreme Threat Models](https://mlsanity.com/paper/2508.20411/) (2025)
- [An Integrated Fusion Framework for Ensemble Learning Leveraging Gradient Boosting and Fuzzy Rule-Based Models](https://mlsanity.com/paper/2511.08077/) (2025)
- [GradCFA: A Hybrid Gradient-Based Counterfactual and Feature Attribution Explanation Algorithm for Local Interpretation of Neural Networks](https://mlsanity.com/paper/2603.15373/) (2026)
- [Modeling Patient Care Trajectories with Transformer Hawkes Processes](https://mlsanity.com/paper/2604.05844/) (2026)
- [NEURON: A Neuro-symbolic System for Grounded Clinical Explainability](https://mlsanity.com/paper/2605.01189/) (2026)
- [A Step Towards Inherently Interpretable Causal Machine Learning Models For Decision Support](https://mlsanity.com/paper/2606.24348/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) models are increasingly finding applications in the field of medicine. Concerns have been raised about the explainability of the decisions that are made by these AI models. In this article, we give a systematic analysis of explainable artificial intelligence (XAI), with a primary focus on models that are currently being used in the field of healthcare. The literature search is conducted following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) standards for relevant work published from 1 January 2012 to 02 February 2022. The review analyzes the prevailing trends in XAI and lays out the major directions in which research is headed. We investigate the why, how, and when of the uses of these XAI models and their implications. We present a comprehensive examination of XAI methodologies as well as an explanation of how a trustworthy AI can be derived from describing AI models for healthcare fields. The discussion of this work will contribute to the formalization of the XAI field.