mlsanityData to 14 Oct 2026

MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images

arXiv 2207.07027 ↗First posted 14 Jul 2022eess.IV cs.CV cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

Multi-modal fusion approaches aim to integrate information from different data sources. Unlike natural datasets, such as in audio-visual applications, where samples consist of "paired" modalities, data in healthcare is often collected asynchronously. Hence, requiring the presence of all modalities for a given sample is not realistic for clinical tasks and significantly limits the size of the dataset during training. In this paper, we propose MedFuse, a conceptually simple yet promising LSTM-based fusion module that can accommodate uni-modal as well as multi-modal input. We evaluate the fusion method and introduce new benchmark results for in-hospital mortality prediction and phenotype classification, using clinical time-series data in the MIMIC-IV dataset and corresponding chest X-ray images in MIMIC-CXR. Compared to more complex multi-modal fusion strategies, MedFuse provides a performance improvement by a large margin on the fully paired test set. It also remains robust across the partially paired test set containing samples with missing chest X-ray images. We release our code for reproducibility and to enable the evaluation of competing models in the future.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 25 of its 56 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

25 of 56 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  6. 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,186
  7. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  8. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  9. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  10. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  11. 2017Multitask learning and benchmarking with clinical time series datacited by 290
  12. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 245
13 more
  1. 2017Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysiscited by 148
  2. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 129
  3. 2019Multimodal Transformer with Multi-View Visual Representation for Image Captioningcited by 86
  4. 2021MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanismscited by 51
  5. 2021End-to-end Multi-modal Video Temporal Groundingcited by 47
  6. 2013Medical Image Fusion: A survey of the state of the artcited by 39
  7. 2021Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transformcited by 38
  8. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  9. 2020Synthesizing the Unseen for Zero-shot Object Detectioncited by 32
  10. 2021Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learningcited by 9
  11. 2021A Multimodal Deep Learning Model for Cardiac Resynchronisation Therapy Response Predictioncited by 7
  12. 2021Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographscited by 5
  13. 2021Towards dynamic multi-modal phenotyping using chest radiographs and physiological datacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 9; 2025: 23; 2026: 15.

54 papers.

Sort:

By area:

  1. 2023Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attentioncited by 25
  2. 2024A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applicationscited by 25
  3. 2025Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Seriescited by 19
  4. 2023LANISTR: Multimodal Learning from Structured and Unstructured Datacited by 11
  5. 2024RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Predictioncited by 10
  6. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?cited by 10
  7. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  8. 2023One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Datacited by 6
  9. 2023Informative Priors Improve the Reliability of Multimodal Clinical Data Classificationcited by 6
  10. 2024Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generationcited by 6
  11. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcitycited by 5
  12. 2025Retrieval-augmented in-context learning for multimodal large language models in disease classificationcited by 5
38 more
  1. 2023Multimodal Representation Learning of Cardiovascular Magnetic Resonance Imagingcited by 4
  2. 2025Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gatecited by 3
  3. 2025PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Datasetcited by 3
  4. 2026AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Taskscited by 3
  5. 2024Towards Predicting Temporal Changes in a Patient's Chest X-ray Images based on Electronic Health Recordscited by 2
  6. 2025MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Taskscited by 2
  7. 2025MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Datacited by 2
  8. 2026An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classificationcited by 2
  9. 2026OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Predictioncited by 2
  10. 2023Few-shot Learning using Data Augmentation and Time-Frequency Transformation for Time Series Classificationcited by 1
  11. 2023Beyond Images: An Integrative Multi-modal Approach to Chest X-Ray Report Generationcited by 1
  12. 2024Deep Learning-Based Noninvasive Screening of Type 2 Diabetes with Chest X-ray Images and Electronic Health Recordscited by 1
  13. 2025Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusioncited by 1
  14. 2025Interpretable Hybrid-Rule Temporal Point Processescited by 1
  15. 2025Small Lesions-aware Bidirectional Multimodal Multiscale Fusion Network for Lung Disease Classificationcited by 1
  16. 2025Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generationcited by 1
  17. 2025RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosiscited by 1
  18. 2025Multimodal Disease Progression Modeling via Spatiotemporal Disentanglement and Multiscale Alignmentcited by 1
  19. 2026LeMoF: Level-guided Multimodal Fusion for Heterogeneous Clinical Datacited by 1
  20. 2026Multimodal Routing for Interpretable, Robust, and Auditable Clinical Predictioncited by 1
  21. 2024Towards Precision Healthcare: Robust Fusion of Time Series and Image Datacited by 0
  22. 2024Multi-modal Masked Siamese Network Improves Chest X-Ray Representation Learningcited by 0
  23. 2024Towards Holistic Disease Risk Prediction using Small Language Modelscited by 0
  24. 2025CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddingscited by 0
  25. 2025Contrastive Cross-Modal Learning for Infusing Chest X-ray Knowledge into ECGscited by 0
  26. 2025Missing-Modality-Aware Graph Neural Network for Cancer Classificationcited by 0
  27. 2025MultiFair: Multimodal Balanced Fairness-Aware Medical Classification with Dual-Level Gradient Modulationcited by 0
  28. 2025BlendFL: Blended Federated Learning for Handling Multimodal Data Heterogeneitycited by 0
  29. 2025Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Processcited by 0
  30. 2025Quantizing Space and Time: Fusing Time Series and Images for Earth Observationcited by 0
  31. 2025Cross-Modal Alignment via Variational Copula Modellingcited by 0
  32. 2025PULSE-ICU: A Pretrained Unified Long-Sequence Encoder for Multi-task Prediction in Intensive Care Unitscited by 0
  33. 2026Heterogeneous Graph Alignment for Joint Reasoning and Interpretabilitycited by 0
  34. 2026Data-Driven Priors for Uncertainty-Aware Risk Prediction of Clinical Deterioration using Multimodal Datacited by 0
  35. 2026EHR-RAGp: Retrieval-Augmented Prototype-Guided Foundation Model for Electronic Health Recordscited by 0
  36. 2026LUNG-KGMM: Knowledge-Guided Multimodal Learning for Lung Cancer Incidence Predictioncited by 0
  37. 2026MOSAIC: Modality-agnostic Spectral Alignment for Federated Image-level Weakly Supervised Tumor Segmentation under Client-specific Missing Modalitiescited by 0
  38. 2026Reliability- and Anatomy-Consistency-Aware Multimodal Learning for Robust Fracture Classification from Bangladeshi Radiographscited by 0

50 most cited of 54 All 54 (JSON)