# HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data

arXiv: https://arxiv.org/abs/2311.09115 · HTML: https://mlsanity.com/paper/2311.09115/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Multimodal Transformer with Multi-View Visual Representation for Image Captioning](https://mlsanity.com/paper/1905.07841/) (2019)
- [TransMed: Transformers Advance Multi-modal Medical Image Classification](https://mlsanity.com/paper/2103.05940/) (2021)
- [Multimodal Optimal Transport-based Co-Attention Transformer with Global Structure Consistency for Survival Prediction](https://mlsanity.com/paper/2306.08330/) (2023)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [Towards Transparent AI Systems: Interpreting Visual Question Answering Models](https://mlsanity.com/paper/1608.08974/) (2016)
- [MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks](https://mlsanity.com/paper/2309.14118/) (2023)
- [Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision](https://mlsanity.com/paper/2304.00379/) (2023)

## Built on by (most cited first, 25 of 25)

- [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025)
- [MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning](https://mlsanity.com/paper/2602.20223/) (2026)
- [Evaluating Explanations Through LLMs: Beyond Traditional User Studies](https://mlsanity.com/paper/2410.17781/) (2024)
- [SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis](https://mlsanity.com/paper/2603.21010/) (2026)
- [Information Bottleneck-Guided Heterogeneous Graph Learning for Interpretable Neurodevelopmental Disorder Diagnosis](https://mlsanity.com/paper/2502.20769/) (2025)
- [MedPatch: Confidence-Guided Multi-Stage Fusion for Multimodal Clinical Data](https://mlsanity.com/paper/2508.09182/) (2025)
- [Explainable Multimodal Regression via Information Decomposition](https://mlsanity.com/paper/2512.22102/) (2025)
- [RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis](https://mlsanity.com/paper/2509.19980/) (2025)
- [Any-to-Any Learning in Computational Pathology via Triplet Multimodal Pretraining](https://mlsanity.com/paper/2505.12711/) (2025)
- [Multimodal Attention-Aware Fusion for Diagnosing Distal Myopathy: Evaluating Model Interpretability and Clinician Trust](https://mlsanity.com/paper/2508.01316/) (2025)
- [Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis](https://mlsanity.com/paper/2509.06831/) (2025)
- [PHASE: Physics-Integrated, Heterogeneity-Aware Surrogates for Scientific Simulations](https://mlsanity.com/paper/2509.23453/) (2025)
- [Imaging-anchored multiomics in cardiovascular disease: integrating cardiac imaging, bulk, single-cell, and spatial transcriptomics](https://mlsanity.com/paper/2601.07871/) (2026)
- [Tissue Classification and Whole-Slide Images Analysis via Modeling of the Tumor Microenvironment and Biological Pathways](https://mlsanity.com/paper/2601.08336/) (2026)
- [Patient-Conditioned Adaptive Offsets for Reliable Diagnosis across Subgroups](https://mlsanity.com/paper/2601.13094/) (2026)
- [Hypernetwork-Based Adaptive Aggregation for Multimodal Multiple-Instance Learning in Predicting Coronary Calcium Debulking](https://mlsanity.com/paper/2601.21479/) (2026)
- [When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion](https://mlsanity.com/paper/2602.23614/) (2026)
- [Clinical-Injection Transformer with Domain-Adapted MAE for Lupus Nephritis Prognosis Prediction](https://mlsanity.com/paper/2603.05535/) (2026)
- [Isotonic Survival Regression: Calibrated Survival Distributions from Deep Cox Models](https://mlsanity.com/paper/2605.16571/) (2026)
- [Multimodality Stacking with Blockwise missing values and application to the PIONeeR biomarkers study for prediction of resistance to immunotherapy](https://mlsanity.com/paper/2605.25050/) (2026)
- [RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities](https://mlsanity.com/paper/2606.05109/) (2026)
- [Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis](https://mlsanity.com/paper/2606.17115/) (2026)
- [Cross-Modal Iteration Distillation for Robust IHD Screening: The IDNet Framework and A New Benchmark](https://mlsanity.com/paper/2606.30027/) (2026)
- [Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention](https://mlsanity.com/paper/2607.19086/) (2026)
- [Multimodal Federated Learning under Dual-Axis Modality Missingness](https://mlsanity.com/paper/2608.09240/) (2026)

## Abstract (arXiv)

Technological advances in medical data collection, such as high-throughput genomic sequencing and digital high-resolution histopathology, have contributed to the rising requirement for multimodal biomedical modelling, specifically for image, tabular and graph data. Most multimodal deep learning approaches use modality-specific architectures that are often trained separately and cannot capture the crucial cross-modal information that motivates the integration of different data sources. This paper presents the Hybrid Early-fusion Attention Learning Network (HEALNet): a flexible multimodal fusion architecture, which a) preserves modality-specific structural information, b) captures the cross-modal interactions and structural information in a shared latent space, c) can effectively handle missing modalities during training and inference, and d) enables intuitive model inspection by learning on the raw data input instead of opaque embeddings. We conduct multimodal survival analysis on Whole Slide Images and Multi-omic data on four cancer datasets from The Cancer Genome Atlas (TCGA). HEALNet achieves state-of-the-art performance compared to other end-to-end trained fusion models, substantially improving over unimodal and multimodal baselines whilst being robust in scenarios with missing modalities.