mlsanityData to 13 Oct 2026

Explainable Deep Learning in Healthcare: A Methodological Survey from an Attribution View

arXiv 2112.02625 ↗First posted 5 Dec 2021cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

The increasing availability of large collections of electronic health record (EHR) data and unprecedented technical advances in deep learning (DL) have sparked a surge of research interest in developing DL based clinical decision support systems for diagnosis, prognosis, and treatment. Despite the recognition of the value of deep learning in healthcare, impediments to further adoption in real healthcare settings remain due to the black-box nature of DL. Therefore, there is an emerging need for interpretable DL, which allows end users to evaluate the model decision making to know whether to accept or reject predictions and recommendations before an action is taken. In this review, we focus on the interpretability of the DL models in healthcare. We start by introducing the methods for interpretability in depth and comprehensively as a methodological reference for future researchers or clinical practitioners in this field. Besides the methods' details, we also include a discussion of advantages and disadvantages of these methods and which scenarios each of them is suitable for, so that interested readers can know how to compare and choose among them for use. Moreover, we discuss how these methods, originally developed for solving general-domain problems, have been adapted and applied to healthcare problems and how they can help physicians better understand these data-driven technologies. Overall, we hope this survey can help researchers and practitioners in both artificial intelligence (AI) and clinical fields understand what methods we have for enhancing the interpretability of their DL models and choose the optimal one accordingly.

Key facts. Cited by 15 papers in the map (data through 13 Oct 2026). Builds on 113 of its 204 references that are in the map. First posted on arXiv 5 Dec 2021.

Builds on

113 of 204 references.

References · 113

  1. 2017Attention Is All You Needcited by 60,106
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,336
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,713
  6. 2017A Unified Approach to Interpreting Model Predictionscited by 6,515
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
  8. 2013Intriguing properties of neural networkscited by 5,137
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,055
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,539
  11. 2017Axiomatic Attribution for Deep Networkscited by 3,368
  12. 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
38 more
  1. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,992
  2. 2015Learning Deep Features for Discriminative Localizationcited by 2,434
  3. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,360
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,282
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,942
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,729
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,710
  9. 2017Learning Important Features Through Propagating Activation Differencescited by 1,455
  10. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,410
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,409
  13. 2019Publicly Available Clinical BERT Embeddingscited by 999
  14. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 999
  15. 2014Striving for Simplicity: The All Convolutional Netcited by 989
  16. 2019Attention is not Explanationcited by 922
  17. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 849
  18. 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
  19. 2016The Mythos of Model Interpretabilitycited by 718
  20. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 676
  21. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 668
  22. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 663
  23. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 594
  24. 2018Deep learning for time series classification: a reviewcited by 575
  25. 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 460
  26. 2018All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneouslycited by 368
  27. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  28. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 351
  29. 2016RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanismcited by 345
  30. 2019The many Shapley values for model explanationcited by 343
  31. 2019On the (In)fidelity and Sensitivity for Explanationscited by 310
  32. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 296
  33. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 291
  34. 2018Consistent Individualized Feature Attribution for Tree Ensemblescited by 282
  35. 2019Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019cited by 273
  36. 2020COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Imagescited by 264
  37. 2018Adversarial Attacks and Defences: A Surveycited by 259
  38. 2019BEHRT: Transformer for Electronic Health Recordscited by 259

50 most cited of 113 All 113 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 1; 2024: 7; 2025: 4.

15 papers.

Sort:

By area:

  1. 2022A Comprehensive Survey on Trustworthy Recommender Systemscited by 32
  2. 2024A Scoping Review of Deep Learning Methods for Photoplethysmography Datacited by 11
  3. 2025There Was Never a Bottleneck in Concept Bottleneck Modelscited by 8
  4. 2021Explainable Artificial Intelligence Methods in Combating Pandemics: A Systematic Reviewcited by 5
  5. 2024ECOR: Explainable CLIP for Object Recognitioncited by 4
  6. 2024Closed-Form Interpretation of Neural Network Classifiers with Symbolic Gradientscited by 3
  7. 2024CauSkelNet: Causal Representation Learning for Human Behaviour Analysiscited by 3
  8. 2024Interpretation of Intracardiac Electrograms Through Textual Representationscited by 2
  9. 2025A Survey on Interpretability in Visual Recognitioncited by 2
  10. 2023Is Task-Agnostic Explainable AI a Myth?cited by 1
  11. 2024Explainable Learning with Gaussian Processescited by 1
  12. 2024Closed-Form Interpretation of Neural Network Latent Spaces with Symbolic Gradientscited by 1
3 more
  1. 2022A Coarse-to-fine Morphological Approach With Knowledge-based Rules and Self-adapting Correction for Lung Nodules Segmentationcited by 0
  2. 2025MindfulLIME: A Stable Solution for Explanations of Machine Learning Models with Enhanced Localization Precision -- A Medical Image Case Studycited by 0
  3. 2025TaylorPODA: A Taylor Expansion-Based Method to Improve Post-Hoc Attributions for Opaque Modelscited by 0