# A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems

arXiv: https://arxiv.org/abs/2502.09849 · HTML: https://mlsanity.com/paper/2502.09849/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 102 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations](https://mlsanity.com/paper/1905.07697/) (2019)
- [Explaining NonLinear Classification Decisions with Deep Taylor Decomposition](https://mlsanity.com/paper/1512.02479/) (2015)
- [Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models](https://mlsanity.com/paper/1612.08468/) (2016)
- [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](https://mlsanity.com/paper/2202.01602/) (2022)
- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](https://mlsanity.com/paper/2110.10790/) (2021)
- [Rethinking Model Evaluation as Narrowing the Socio-Technical Gap](https://mlsanity.com/paper/2306.03100/) (2023)
- ["The Human Body is a Black Box": Supporting Clinical Decision-Making with Deep Learning](https://mlsanity.com/paper/1911.08089/) (2019)
- [Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation](https://mlsanity.com/paper/2010.15728/) (2020)
- [Ignore, Trust, or Negotiate: Understanding Clinician Acceptance of AI-Based Treatment Recommendations in Health Care](https://mlsanity.com/paper/2302.00096/) (2023)
- [Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis](https://mlsanity.com/paper/2309.12368/) (2023)
- [Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorations](https://mlsanity.com/paper/2302.10671/) (2023)
- [Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma](https://mlsanity.com/paper/2303.12806/) (2023)
- [An Explainable CNN Approach for Medical Codes Prediction from Clinical Text](https://mlsanity.com/paper/2101.11430/) (2021)
- [Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes](https://mlsanity.com/paper/2302.05752/) (2023)
- [Three-stage intelligent support of clinical decision making for higher trust, validity, and explainability](https://mlsanity.com/paper/2007.12870/) (2020)
- [An Incremental Explanation of Inference in Hybrid Bayesian Networks for Increasing Model Trustworthiness and Supporting Clinical Decision Making](https://mlsanity.com/paper/2003.02599/) (2020)

## Built on by (most cited first, 3 of 3)

- [RECA-PD: A Robust Explainable Cross-Attention Method for Speech-based Parkinson's Disease Classification](https://mlsanity.com/paper/2507.03594/) (2025)
- [Secure Diagnostics: Adversarial Robustness Meets Clinical Interpretability](https://mlsanity.com/paper/2504.05483/) (2025)
- [Divergent Realities: A Comparative Analysis of Human Expert vs. Artificial Intelligence Based Generation and Evaluation of Treatment Plans in Dermatology](https://mlsanity.com/paper/2507.05716/) (2025)

## Abstract (arXiv)

Explainable Artificial Intelligence (XAI) is essential for the transparency and clinical adoption of Clinical Decision Support Systems (CDSS). However, the real-world effectiveness of existing XAI methods remains limited and is inconsistently evaluated. This study conducts a systematic PRISMA-guided survey of 31 human-centered evaluations (HCE) of XAI applied to CDSS, classifying them by XAI methodology, evaluation design, and adoption barrier. Our findings reveal that most existing studies employ post-hoc, model-agnostic approaches such as SHAP and Grad-CAM, typically assessed through small-scale clinician studies. The results show that over 80% of the studies adopt post-hoc, model-agnostic approaches such as SHAP and Grad-CAM, and that clinician sample sizes remain below 25 participants. The findings indicate that explanations generally improve clinician trust and diagnostic confidence, but frequently increase cognitive load and exhibit misalignment with domain reasoning processes. To bridge these gaps, we propose a stakeholder-centric evaluation framework that integrates socio-technical principles and human-computer interaction to guide the future development of clinically viable and trustworthy XAI-based CDSS.