# Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond

arXiv: https://arxiv.org/abs/2102.01998 · HTML: https://mlsanity.com/paper/2102.01998/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 201 references that are in the map.
- First posted on arXiv 3 Feb 2021.

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Learning Important Features Through Propagating Activation Differences](https://mlsanity.com/paper/1704.02685/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Object Detection with Deep Learning: A Review](https://mlsanity.com/paper/1807.05511/) (2018)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Methods for Interpreting and Understanding Deep Neural Networks](https://mlsanity.com/paper/1706.07979/) (2017)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest X-Ray Images](https://mlsanity.com/paper/2003.09871/) (2020)
- [Techniques for Interpretable Machine Learning](https://mlsanity.com/paper/1808.00033/) (2018)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)

## Built on by (most cited first, 30 of 76)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [Transparency of Deep Neural Networks for Medical Image Analysis: A Review of Interpretability Methods](https://mlsanity.com/paper/2111.02398/) (2021)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvement](https://mlsanity.com/paper/2203.08008/) (2022)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [Guidelines and Evaluation of Clinical Explainable AI in Medical Image Analysis](https://mlsanity.com/paper/2202.10553/) (2022)
- [A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?](https://mlsanity.com/paper/2304.04780/) (2023)
- [A survey, review, and future trends of skin lesion segmentation and classification](https://mlsanity.com/paper/2208.12232/) (2022)
- [Data Harmonisation for Information Fusion in Digital Healthcare: A State-of-the-Art Systematic Review, Meta-Analysis and Future Research Directions](https://mlsanity.com/paper/2201.06505/) (2022)
- [Importance-Aware Adaptive Dataset Distillation](https://mlsanity.com/paper/2401.15863/) (2024)
- [Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlooks](https://mlsanity.com/paper/2410.02331/) (2024)
- [A Comprehensive Survey on Self-Interpretable Neural Networks](https://mlsanity.com/paper/2501.15638/) (2025)
- [Measuring Cross-Modal Interactions in Multimodal Models](https://mlsanity.com/paper/2412.15828/) (2024)
- [Federated Multi-organ Segmentation with Inconsistent Labels](https://mlsanity.com/paper/2206.07156/) (2022)
- [Swin Deformable Attention U-Net Transformer (SDAUT) for Explainable Fast MRI](https://mlsanity.com/paper/2207.02390/) (2022)
- [Human Treelike Tubular Structure Segmentation: A Comprehensive Review and Future Perspectives](https://mlsanity.com/paper/2207.11203/) (2022)
- [Remote patient monitoring using artificial intelligence: Current state, applications, and challenges](https://mlsanity.com/paper/2301.10009/) (2023)
- [FUTURE-AI: Guiding Principles and Consensus Recommendations for Trustworthy Artificial Intelligence in Medical Imaging](https://mlsanity.com/paper/2109.09658/) (2021)
- [Explainable AI For COVID-19 CT Classifiers: An Initial Comparison Study](https://mlsanity.com/paper/2104.14506/) (2021)
- [Designing Interpretable ML System to Enhance Trust in Healthcare: A Systematic Review to Proposed Responsible Clinician-AI-Collaboration Framework](https://mlsanity.com/paper/2311.11055/) (2023)
- [Holding AI to Account: Challenges for the Delivery of Trustworthy AI in Healthcare](https://mlsanity.com/paper/2211.16444/) (2022)
- [A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future](https://mlsanity.com/paper/2412.14056/) (2024)
- [Explainable, Domain-Adaptive, and Federated Artificial Intelligence in Medicine](https://mlsanity.com/paper/2211.09317/) (2022)
- [Large-Kernel Attention for 3D Medical Image Segmentation](https://mlsanity.com/paper/2207.11225/) (2022)
- [CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection](https://mlsanity.com/paper/2506.01890/) (2025)
- [Robust Weakly Supervised Learning for COVID-19 Recognition Using Multi-Center CT Images](https://mlsanity.com/paper/2112.04984/) (2021)
- [Explainable COVID-19 Infections Identification and Delineation Using Calibrated Pseudo Labels](https://mlsanity.com/paper/2202.07422/) (2022)

## Abstract (arXiv)

Explainable Artificial Intelligence (XAI) is an emerging research topic of machine learning aimed at unboxing how AI systems' black-box choices are made. This research field inspects the measures and models involved in decision-making and seeks solutions to explain them explicitly. Many of the machine learning algorithms can not manifest how and why a decision has been cast. This is particularly true of the most popular deep neural network approaches currently in use. Consequently, our confidence in AI systems can be hindered by the lack of explainability in these black-box models. The XAI becomes more and more crucial for deep learning powered applications, especially for medical and healthcare studies, although in general these deep neural networks can return an arresting dividend in performance. The insufficient explainability and transparency in most existing AI systems can be one of the major reasons that successful implementation and integration of AI tools into routine clinical practice are uncommon. In this study, we first surveyed the current progress of XAI and in particular its advances in healthcare applications. We then introduced our solutions for XAI leveraging multi-modal and multi-centre data fusion, and subsequently validated in two showcases following real clinical scenarios. Comprehensive quantitative and qualitative analyses can prove the efficacy of our proposed XAI solutions, from which we can envisage successful applications in a broader range of clinical questions.