# Explainable Artificial Intelligence for Human Decision-Support System in Medical Domain

arXiv: https://arxiv.org/abs/2105.02357 · HTML: https://mlsanity.com/paper/2105.02357/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 52 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 14 of 14)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?](https://mlsanity.com/paper/2005.01831/) (2020)
- [Explainable Machine Learning for Scientific Insights and Discoveries](https://mlsanity.com/paper/1905.08883/) (2019)
- [What do we need to build explainable AI systems for the medical domain?](https://mlsanity.com/paper/1712.09923/) (2017)
- [Explainable Deep Learning: A Field Guide for the Uninitiated](https://mlsanity.com/paper/2004.14545/) (2020)
- [Visual Analytics for Explainable Deep Learning](https://mlsanity.com/paper/1804.02527/) (2018)
- [Towards the Augmented Pathologist: Challenges of Explainable-AI in Digital Pathology](https://mlsanity.com/paper/1712.06657/) (2017)
- [Medical Deep Learning -- A systematic Meta-Review](https://mlsanity.com/paper/2010.14881/) (2020)
- [Explainable AI without Interpretable Model](https://mlsanity.com/paper/2009.13996/) (2020)

## Built on by (most cited first, 12 of 12)

- [A Survey on Explainable Artificial Intelligence for Cybersecurity](https://mlsanity.com/paper/2303.12942/) (2023)
- [Beyond Known Reality: Exploiting Counterfactual Explanations for Medical Research](https://mlsanity.com/paper/2307.02131/) (2023)
- [How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps](https://mlsanity.com/paper/2404.16042/) (2024)
- [Contextual Importance and Utility: aTheoretical Foundation](https://mlsanity.com/paper/2202.07292/) (2022)
- [X-Ray-CoT: Interpretable Chest X-ray Diagnosis with Vision-Language Models via Chain-of-Thought Reasoning](https://mlsanity.com/paper/2508.12455/) (2025)
- [Segmentation and Smoothing Affect Explanation Quality More Than the Choice of Perturbation-based XAI Method for Image Explanations](https://mlsanity.com/paper/2409.04116/) (2024)
- [Informed Deep Abstaining Classifier: Investigating noise-robust training for diagnostic decision support systems](https://mlsanity.com/paper/2410.21014/) (2024)
- [XAI for In-hospital Mortality Prediction via Multimodal ICU Data](https://mlsanity.com/paper/2312.17624/) (2023)
- [Predicting Space Tourism Demand Using Explainable AI](https://mlsanity.com/paper/2503.03113/) (2025)
- [Feature-Guided Neighbor Selection for Non-Expert Evaluation of Model Predictions](https://mlsanity.com/paper/2507.06029/) (2025)
- [Analysis of Invasive Breast Cancer in Mammograms Using YOLO, Explainability, and Domain Adaptation](https://mlsanity.com/paper/2512.00129/) (2025)
- [A User-Centric Analysis of Explainability in AI-Based Medical Image Diagnosis](https://mlsanity.com/paper/2605.02903/) (2026)

## Abstract (arXiv)

In the present paper we present the potential of Explainable Artificial Intelligence methods for decision-support in medical image analysis scenarios. With three types of explainable methods applied to the same medical image data set our aim was to improve the comprehensibility of the decisions provided by the Convolutional Neural Network (CNN). The visual explanations were provided on in-vivo gastral images obtained from a Video capsule endoscopy (VCE), with the goal of increasing the health professionals' trust in the black box predictions. We implemented two post-hoc interpretable machine learning methods LIME and SHAP and the alternative explanation approach CIU, centered on the Contextual Value and Utility (CIU). The produced explanations were evaluated using human evaluation. We conducted three user studies based on the explanations provided by LIME, SHAP and CIU. Users from different non-medical backgrounds carried out a series of tests in the web-based survey setting and stated their experience and understanding of the given explanations. Three user groups (n=20, 20, 20) with three distinct forms of explanations were quantitatively analyzed. We have found that, as hypothesized, the CIU explainable method performed better than both LIME and SHAP methods in terms of increasing support for human decision-making as well as being more transparent and thus understandable to users. Additionally, CIU outperformed LIME and SHAP by generating explanations more rapidly. Our findings suggest that there are notable differences in human decision-making between various explanation support settings. In line with that, we present three potential explainable methods that can with future improvements in implementation be generalized on different medical data sets and can provide great decision-support for medical experts.