# Multimodal Explainability via Latent Shift applied to COVID-19 stratification

arXiv: https://arxiv.org/abs/2212.14084 · HTML: https://mlsanity.com/paper/2212.14084/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 54 references that are in the map.
- First posted on arXiv 28 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [PadChest: A large chest x-ray image dataset with multi-label annotated reports](https://mlsanity.com/paper/1901.07441/) (2019)
- [Multimodal Explanations: Justifying Decisions and Pointing to the Evidence](https://mlsanity.com/paper/1802.08129/) (2018)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)
- [TorchXRayVision: A library of chest X-ray datasets and models](https://mlsanity.com/paper/2111.00595/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [A Survey of Deep Learning for Scientific Discovery](https://mlsanity.com/paper/2003.11755/) (2020)
- [Checklist for responsible deep learning modeling of medical images based on COVID-19 detection studies](https://mlsanity.com/paper/2012.08333/) (2020)
- [AIforCOVID: predicting the clinical outcomes in patients with COVID-19 applying AI to chest-X-rays. An Italian multicentre study](https://mlsanity.com/paper/2012.06531/) (2020)
- [Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomes](https://mlsanity.com/paper/2204.03772/) (2022)

## Built on by (most cited first, 16 of 16)

- [A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications](https://mlsanity.com/paper/2408.02686/) (2024)
- [Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications](https://mlsanity.com/paper/2411.03782/) (2024)
- [Multi-Dataset Multi-Task Learning for COVID-19 Prognosis](https://mlsanity.com/paper/2405.13771/) (2024)
- [A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1](https://mlsanity.com/paper/2312.17506/) (2023)
- [Class Balancing Diversity Multimodal Ensemble for Alzheimer's Disease Diagnosis and Early Detection](https://mlsanity.com/paper/2410.10374/) (2024)
- [Whole-Body Image-to-Image Translation for a Virtual Scanner in a Healthcare Digital Twin](https://mlsanity.com/paper/2503.15555/) (2025)
- [Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation](https://mlsanity.com/paper/2505.01091/) (2025)
- [Lesion-Aware Generative Artificial Intelligence for Virtual Contrast-Enhanced Mammography in Breast Cancer](https://mlsanity.com/paper/2505.03018/) (2025)
- [Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer](https://mlsanity.com/paper/2505.01390/) (2025)
- [Multi-modal Evidential Fusion Network for Trustworthy PET/CT Tumor Segmentation](https://mlsanity.com/paper/2406.18327/) (2024)
- [Multi-stage intermediate fusion for multimodal learning to classify non-small cell lung cancer subtypes from CT and PET](https://mlsanity.com/paper/2501.12425/) (2025)
- [Beyond a Single Mode: GAN Ensembles for Diverse Medical Data Generation](https://mlsanity.com/paper/2503.24258/) (2025)
- [Cross-dataset Multivariate Time-series Model for Parkinson's Diagnosis via Keyboard Dynamics](https://mlsanity.com/paper/2510.15950/) (2025)
- [Handling Missing Modalities in Multimodal Survival Prediction for Non-Small Cell Lung Cancer](https://mlsanity.com/paper/2601.10386/) (2026)
- [Cross Modality Image Translation In Medical Imaging Using Generative Frameworks](https://mlsanity.com/paper/2605.13686/) (2026)
- [Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languages](https://mlsanity.com/paper/2505.01096/) (2025)

## Abstract (arXiv)

We are witnessing a widespread adoption of artificial intelligence in healthcare. However, most of the advancements in deep learning in this area consider only unimodal data, neglecting other modalities. Their multimodal interpretation necessary for supporting diagnosis, prognosis and treatment decisions. In this work we present a deep architecture, which jointly learns modality reconstructions and sample classifications using tabular and imaging data. The explanation of the decision taken is computed by applying a latent shift that, simulates a counterfactual prediction revealing the features of each modality that contribute the most to the decision and a quantitative score indicating the modality importance. We validate our approach in the context of COVID-19 pandemic using the AIforCOVID dataset, which contains multimodal data for the early identification of patients at risk of severe outcome. The results show that the proposed method provides meaningful explanations without degrading the classification performance.