mlsanityData to 14 Oct 2026

Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis

arXiv 2202.10553 ↗First posted 16 Feb 2022cs.LG cs.AI cs.CVcited by 35

Not on the map. Why?

Abstract (arXiv)

Explainable artificial intelligence (XAI) is essential for enabling clinical users to get informed decision support from AI and comply with evidence-based medical practice. Applying XAI in clinical settings requires proper evaluation criteria to ensure the explanation technique is both technically sound and clinically useful, but specific support is lacking to achieve this goal. To bridge the research gap, we propose the Clinical XAI Guidelines that consist of five criteria a clinical XAI needs to be optimized for. The guidelines recommend choosing an explanation form based on Guideline 1 (G1) Understandability and G2 Clinical relevance. For the chosen explanation form, its specific XAI technique should be optimized for G3 Truthfulness, G4 Informative plausibility, and G5 Computational efficiency. Following the guidelines, we conducted a systematic evaluation on a novel problem of multi-modal medical image explanation with two clinical tasks, and proposed new evaluation metrics accordingly. Sixteen commonly-used heatmap XAI techniques were evaluated and found to be insufficient for clinical use due to their failure in G3 and G4. Our evaluation demonstrated the use of Clinical XAI Guidelines to support the design and evaluation of clinically viable XAI.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 54 of its 121 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

54 of 121 references.

References · 54

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  11. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,552
  12. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
38 more
  1. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  2. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  3. 2018Sanity Checks for Saliency Mapscited by 1,156
  4. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  5. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  6. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  7. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 737
  8. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  9. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  10. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  11. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  12. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  13. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  14. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 373
  15. 2018On the Robustness of Interpretability Methodscited by 314
  16. 2019On the (In)fidelity and Sensitivity for Explanationscited by 310
  17. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 297
  18. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  19. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  20. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 190
  21. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  22. 2018Visual Interpretability for Deep Learning: a Surveycited by 155
  23. 2020Debugging Tests for Model Explanationscited by 147
  24. 2020Reliable Post hoc Explanations: Modeling Uncertainty in Explainabilitycited by 125
  25. 2020Explainable deep learning models in medical image analysiscited by 111
  26. 2019Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Makingcited by 109
  27. 2019Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approachescited by 101
  28. 2021Do Feature Attribution Methods Correctly Attribute Features?cited by 98
  29. 2020Shapley explainability on the data manifoldcited by 88
  30. 2021Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyondcited by 76
  31. 2019U-CAM: Visual Explanation using Uncertainty based Class Activation Mapscited by 36
  32. 2020AI Research Considerations for Human Existential Safety (ARCHES)cited by 33
  33. 2020Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imagingcited by 32
  34. 2018Deep learning cardiac motion analysis for human survival predictioncited by 31
  35. 2021Algorithm Fairness in AI for Medicine and Healthcarecited by 21
  36. 2022Data Harmonisation for Information Fusion in Digital Healthcare: A State-of-the-Art Systematic Review, Meta-Analysis and Future Research Directionscited by 19
  37. 2019InfoMask: Masked Variational Latent Representation to Localize Chest Diseasecited by 15
  38. 2021One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Imagescited by 11

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 9; 2024: 8; 2025: 6; 2026: 9.

35 papers.

Sort:

By area:

  1. 2024XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimizationcited by 20
  2. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  3. 2024Explainable Artificial Intelligence for Medical Applications: A Reviewcited by 12
  4. 2023MProtoNet: A Case-Based Interpretable Model for Brain Tumor Classification with 3D Multi-parametric Magnetic Resonance Imagingcited by 11
  5. 2023Why is plausibility surprisingly problematic as an XAI criterion?cited by 9
  6. 2024DiffExplainer: Unveiling Black Box Models Via Counterfactual Generationcited by 5
  7. 2024Aggregated Attributions for Explanatory Analysis of 3D Segmentation Modelscited by 5
  8. 2024Understanding How Blind Users Handle Object Recognition Errors: Strategies and Challengescited by 5
  9. 2023Invisible Users: Uncovering End-Users' Requirements for Explainable AI via Explanation Forms and Goalscited by 4
  10. 2024PIPNet3D: Interpretable Detection of Alzheimer in MRI Scanscited by 3
  11. 2025Transparent AI: The Case for Interpretability and Explainabilitycited by 3
  12. 2022Transcending XAI Algorithm Boundaries through End-User-Inspired Designcited by 2
23 more
  1. 2023A novel framework employing deep multi-attention channels network for the autonomous detection of metastasizing cells through fluorescence microscopycited by 2
  2. 2026Beyond Attention Heatmaps: How to Get Better Explanations for Multiple Instance Learning Models in Histopathologycited by 2
  3. 2023Explainable Image Quality Assessment for Medical Imagingcited by 1
  4. 2023End-To-End Prediction of Knee Osteoarthritis Progression With Multi-Modal Transformerscited by 1
  5. 2025SpecReX: Explainable AI for Raman Spectroscopycited by 1
  6. 2026Ranking XAI Methods for Head and Neck Cancer Outcome Predictioncited by 1
  7. 2026M3Net: A Macro-to-Meso-to-Micro Clinical-inspired Hierarchical 3D Network for Pulmonary Nodule Classificationcited by 1
  8. 2022Adversarial Attacks and Defences for Skin Cancer Classificationcited by 0
  9. 2022Analysis of Explainable Artificial Intelligence Methods on Medical Image Classificationcited by 0
  10. 2023AI-based analysis of super-resolution microscopy: Biological discovery in the absence of ground truthcited by 0
  11. 2023SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology predictioncited by 0
  12. 2023Towards Interpretable Classification of Leukocytes based on Deep Learningcited by 0
  13. 2024Explainable AI: Definition and attributes of a good explanation for health AIcited by 0
  14. 2025PointExplainer: Towards Transparent Parkinson's Disease Diagnosiscited by 0
  15. 2025Transferring Visual Explainability of Self-Explaining Models to Prediction-Only Models without Additional Trainingcited by 0
  16. 2025Towards a Transparent and Interpretable AI Model for Medical Image Classificationscited by 0
  17. 2025CoPA: Hierarchical Concept Prompting and Aggregating Network for Explainable Diagnosiscited by 0
  18. 2026Interpretability and Individuality in Knee MRI: Patient-Specific Radiomic Fingerprint with Reconstructed Healthy Personascited by 0
  19. 2026Bridging visual saliency and large language models for explainable deep learning in medical imagingcited by 0
  20. 2026Quantifying Explainable AI-introduced signal noise on ECG data with Spectral Entropycited by 0
  21. 2026Locally Private Online Quantile Regression: Estimation and Inferencecited by 0
  22. 2026SMILE: Self-Explainable Multimodal Information Bottleneck for Medical Diagnosiscited by 0
  23. 2026Architecture-aware Robustness Evaluation of Explainable Deep Learning for Breast Cancer Diagnosiscited by 0