mlsanityData to 7 Oct 2026

Multimodal Explainability via Latent Shift applied to COVID-19 stratification

arXiv 2212.14084 ↗First posted 28 Dec 2022cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We are witnessing a widespread adoption of artificial intelligence in healthcare. However, most of the advancements in deep learning in this area consider only unimodal data, neglecting other modalities. Their multimodal interpretation necessary for supporting diagnosis, prognosis and treatment decisions. In this work we present a deep architecture, which jointly learns modality reconstructions and sample classifications using tabular and imaging data. The explanation of the decision taken is computed by applying a latent shift that, simulates a counterfactual prediction revealing the features of each modality that contribute the most to the decision and a quantitative score indicating the modality importance. We validate our approach in the context of COVID-19 pandemic using the AIforCOVID dataset, which contains multimodal data for the early identification of patients at risk of severe outcome. The results show that the proposed method provides meaningful explanations without degrading the classification performance.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 18 of its 54 references that are in the map. First posted on arXiv 28 Dec 2022.

Builds on

18 of 54 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  4. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  6. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  7. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  8. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
  9. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 403
  10. 2019PadChest: A large chest x-ray image dataset with multi-label annotated reportscited by 308
  11. 2018Multimodal Explanations: Justifying Decisions and Pointing to the Evidencecited by 179
  12. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
6 more
  1. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  2. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  3. 2020A Survey of Deep Learning for Scientific Discoverycited by 36
  4. 2020Checklist for responsible deep learning modeling of medical images based on COVID-19 detection studiescited by 15
  5. 2020AIforCOVID: predicting the clinical outcomes in patients with COVID-19 applying AI to chest-X-rays. An Italian multicentre studycited by 13
  6. 2022Multi-objective optimization determines when, which and how to fuse deep networks: an application to predict COVID-19 outcomescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 8; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applicationscited by 25
  2. 2024Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applicationscited by 13
  3. 2024Multi-Dataset Multi-Task Learning for COVID-19 Prognosiscited by 11
  4. 2023A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1cited by 7
  5. 2024Class Balancing Diversity Multimodal Ensemble for Alzheimer's Disease Diagnosis and Early Detectioncited by 5
  6. 2025Whole-Body Image-to-Image Translation for a Virtual Scanner in a Healthcare Digital Twincited by 5
  7. 2025Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generationcited by 3
  8. 2025Lesion-Aware Generative Artificial Intelligence for Virtual Contrast-Enhanced Mammography in Breast Cancercited by 3
  9. 2025Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancercited by 2
  10. 2024Multi-modal Evidential Fusion Network for Trustworthy PET/CT Tumor Segmentationcited by 1
  11. 2025Multi-stage intermediate fusion for multimodal learning to classify non-small cell lung cancer subtypes from CT and PETcited by 1
  12. 2025Beyond a Single Mode: GAN Ensembles for Diverse Medical Data Generationcited by 1
4 more
  1. 2025Cross-dataset Multivariate Time-series Model for Parkinson's Diagnosis via Keyboard Dynamicscited by 1
  2. 2026Handling Missing Modalities in Multimodal Survival Prediction for Non-Small Cell Lung Cancercited by 1
  3. 2026Cross Modality Image Translation In Medical Imaging Using Generative Frameworkscited by 1
  4. 2025Evaluating Vision Language Model Adaptations for Radiology Report Generation in Low-Resource Languagescited by 0