mlsanityData to 7 Oct 2026

Explainable artificial intelligence (XAI) in deep learning-based medical image analysis

arXiv 2107.10912 ↗First posted 22 Jul 2021eess.IV cs.CVcited by 147

Not on the map. Why?

Abstract (arXiv)

With an increase in deep learning-based methods, the call for explainability of such methods grows, especially in high-stakes decision making areas such as medical image analysis. This survey presents an overview of eXplainable Artificial Intelligence (XAI) used in deep learning-based medical image analysis. A framework of XAI criteria is introduced to classify deep learning-based medical image analysis methods. Papers on XAI techniques in medical image analysis are then surveyed and categorized according to the framework and according to anatomical location. The paper concludes with an outlook of future opportunities for XAI in medical image analysis.

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Builds on 94 of its 297 references that are in the map. First posted on arXiv 22 Jul 2021.

Builds on

94 of 297 references.

References · 94

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  7. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  8. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  12. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
38 more
  1. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,182
  2. 2018Sanity Checks for Saliency Mapscited by 1,149
  3. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  4. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  5. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  6. 2017Dynamic Routing Between Capsulescited by 689
  7. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  8. 2014Deep metric learning using Triplet networkcited by 421
  9. 2016Ladder Variational Autoencoderscited by 315
  10. 2019Interpretable machine learning: definitions, methods, and applicationscited by 303
  11. 2018Attention Gated Networks: Learning to Leverage Salient Regions in Medical Imagescited by 271
  12. 2017On the Automatic Generation of Medical Imaging Reportscited by 215
  13. 2019Causality matters in medical imagingcited by 193
  14. 2018Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysiscited by 178
  15. 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
  16. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  17. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 101
  18. 2020On Interpretability of Artificial Neural Networks: A Surveycited by 88
  19. 2018Learn To Pay Attentioncited by 83
  20. 2017Detecting Statistical Interactions from Neural Network Weightscited by 80
  21. 2019Neural Network Attributions: A Causal Perspectivecited by 77
  22. 2021Going Deeper through the Gleason Scoring Scale: An Automatic end-to-end System for Histology Prostate Grading and Cribriform Pattern Detectioncited by 66
  23. 2019Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichmentcited by 60
  24. 2017MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Networkcited by 56
  25. 2017Thoracic Disease Identification and Localization with Limited Supervisioncited by 55
  26. 2018Learning to Summarize Radiology Findingscited by 54
  27. 2017Visual Feature Attribution using Wasserstein GANscited by 47
  28. 2017Deep Lesion Graphs in the Wild: Relationship Learning and Organization of Significant Radiology Image Findings in a Diverse Large-scale Lesion Databasecited by 42
  29. 2020Understanding Graph Isomorphism Network for rs-fMRI Functional Connectivity Analysiscited by 41
  30. 2020Open Source Software for Efficient and Transparent Reviewscited by 34
  31. 2019A Mutual Bootstrapping Model for Automated Skin Lesion Segmentation and Classificationcited by 32
  32. 2019Resolving challenges in deep learning-based analyses of histopathological images using explanation methodscited by 32
  33. 2020Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imagingcited by 32
  34. 2019Global and Local Interpretability for Cardiac MRI Classificationcited by 29
  35. 2018An Interpretable Deep Hierarchical Semantic Convolutional Neural Network for Lung Nodule Malignancy Classificationcited by 25
  36. 2019Testing the robustness of attribution methods for convolutional neural networks in MRI-based Alzheimer's disease classificationcited by 25
  37. 2019DRvertGRADUATE: uncertainty-aware deep learning-based diabetic retinopathy grading in eye fundus imagescited by 25
  38. 2019Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional MRI using layer-wise relevance propagationcited by 21

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 13; 2023: 37; 2024: 46; 2025: 24; 2026: 25.

147 papers.

Sort:

By area:

  1. 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
  2. 2024DriveCoT: Integrating Chain-of-Thought Reasoning with End-to-End Drivingcited by 67
  3. 2022Trustworthy clinical AI solutions: a unified review of uncertainty quantification in deep learning models for medical image analysiscited by 62
  4. 2023Learning Bottleneck Concepts in Image Classificationcited by 49
  5. 2023Domain Generalization for Medical Image Analysis: A Reviewcited by 47
  6. 2023Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Modelscited by 40
  7. 2022Explainable Deep Learning Methods in Medical Image Classification: A Surveycited by 31
  8. 2021Explainable Medical Imaging AI Needs Human-Centered Design: Guidelines and Evidence from a Systematic Reviewcited by 30
  9. 2022SAFARI: Versatile and Efficient Evaluations for Robustness of Interpretabilitycited by 28
  10. 2022Evaluating Feature Attribution Methods in the Image Domaincited by 19
  11. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  12. 2023The role of causality in explainable artificial intelligencecited by 17
38 more
  1. 2023Assessing Fidelity in XAI post-hoc techniques: A Comparative Study with Ground Truth Explanations Datasetscited by 15
  2. 2023DreaMR: Diffusion-driven Counterfactual Explanation for Functional MRIcited by 14
  3. 2024A comprehensive study on fidelity metrics for XAIcited by 14
  4. 2023A scoping review on multimodal deep learning in biomedical images and textscited by 13
  5. 2024Explainable Artificial Intelligence for Medical Applications: A Reviewcited by 12
  6. 2023A Framework for Interpretability in Machine Learning for Medical Imagingcited by 9
  7. 2022INSightR-Net: Interpretable Neural Network for Regression using Similarity-based Comparisons to Prototypical Examplescited by 8
  8. 2023On the Impact of Knowledge Distillation for Model Interpretabilitycited by 8
  9. 2022Attention-based Interpretable Regression of Gene Expression in Histologycited by 7
  10. 2022Deep Learning Generates Synthetic Cancer Histology for Explainability and Educationcited by 7
  11. 2023Slideflow: Deep Learning for Digital Histopathology with Real-Time Whole-Slide Visualizationcited by 7
  12. 2024Evidential Concept Embedding Models: Towards Reliable Concept Explanations for Skin Disease Diagnosiscited by 7
  13. 2024Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Featurescited by 7
  14. 2023MRxaI: Black-Box Explainability for Image Classifiers in a Medical Settingcited by 6
  15. 2024Towards Multi-dimensional Explanation Alignment for Medical Classificationcited by 6
  16. 2023EvalAttAI: A Holistic Approach to Evaluating Attribution Maps in Robust and Non-Robust Modelscited by 5
  17. 2023Understanding Calibration of Deep Neural Networks for Medical Image Classificationcited by 5
  18. 2024CONFINE: Conformal Prediction for Interpretable Neural Networkscited by 5
  19. 2024DWARF: Disease-weighted network for attention map refinementcited by 5
  20. 2024Aggregated Attributions for Explanatory Analysis of 3D Segmentation Modelscited by 5
  21. 2025Here Comes the Explanation: A Shapley Perspective on Multi-contrast Medical Image Segmentationcited by 5
  22. 2023A novel approach to generate datasets with XAI ground truth to evaluate image modelscited by 4
  23. 2023AI-in-the-Loop -- The impact of HMI in AI-based Applicationcited by 4
  24. 2024CAPE: CAM as a Probabilistic Ensemble for Enhanced DNN Interpretationcited by 4
  25. 2024Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretationcited by 4
  26. 2025Interpretable Retinal Disease Prediction Using Biology-Informed Heterogeneous Graph Representationscited by 4
  27. 2025Learning Concept-Driven Logical Rules for Interpretable and Generalizable Medical Image Classificationcited by 4
  28. 2025X-Ray-CoT: Interpretable Chest X-ray Diagnosis with Vision-Language Models via Chain-of-Thought Reasoningcited by 4
  29. 2023sMRI-PatchNet: A novel explainable patch-based deep learning network for Alzheimer's disease diagnosis and discriminative atrophy localisation with Structural MRIcited by 3
  30. 2024PIPNet3D: Interpretable Detection of Alzheimer in MRI Scanscited by 3
  31. 2024Transferring Domain Knowledge with (X)AI-Based Learning Systemscited by 3
  32. 2024Towards consistency of rule-based explainer and black box model -- fusion of rule induction and XAI-based feature importancecited by 3
  33. 2024IBO: Inpainting-Based Occlusion to Enhance Explainable Artificial Intelligence Evaluation in Histopathologycited by 3
  34. 2022Reducing Annotation Need in Self-Explanatory Models for Lung Nodule Diagnosiscited by 2
  35. 2022Automatic Infectious Disease Classification Analysis with Concept Discoverycited by 2
  36. 2022Explainable AI over the Internet of Things (IoT): Overview, State-of-the-Art and Future Directionscited by 2
  37. 2022Weakly-supervised detection of AMD-related lesions in color fundus images using explainable deep learningcited by 2
  38. 2023Gender, Smoking History and Age Prediction from Laryngeal Imagescited by 2

50 most cited of 147 All 147 (JSON)