# HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling

arXiv: https://arxiv.org/abs/2403.13319 · HTML: https://mlsanity.com/paper/2403.13319/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 81 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1710.10916/) (2017)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation](https://mlsanity.com/paper/1904.07773/) (2019)
- [Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data](https://mlsanity.com/paper/2303.14080/) (2023)
- [Deep Meta Functionals for Shape Representation](https://mlsanity.com/paper/1908.06277/) (2019)
- [Evaluating the Impact of Intensity Normalization on MR Image Synthesis](https://mlsanity.com/paper/1812.04652/) (2018)
- [Principled Weight Initialization for Hypernetworks](https://mlsanity.com/paper/2312.08399/) (2023)
- [LIMITR: Leveraging Local Information for Medical Image-Text Representation](https://mlsanity.com/paper/2303.11755/) (2023)
- [End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification](https://mlsanity.com/paper/1810.00523/) (2018)
- [HyperTab: Hypernetwork Approach for Deep Learning on Small Tabular Datasets](https://mlsanity.com/paper/2304.03543/) (2023)
- [Hypernetwork-Based Adaptive Image Restoration](https://mlsanity.com/paper/2206.05970/) (2022)
- [Analysing race and sex bias in brain age prediction](https://mlsanity.com/paper/2309.10835/) (2023)
- [A hybrid machine learning/deep learning COVID-19 severity predictive model from CT images and clinical data](https://mlsanity.com/paper/2105.06141/) (2021)
- [Sample-Specific Debiasing for Better Image-Text Models](https://mlsanity.com/paper/2304.13181/) (2023)

## Built on by (most cited first, 15 of 15)

- [Unleashing the Power of Image-Tabular Self-Supervised Learning via Breaking Cross-Tabular Barriers](https://mlsanity.com/paper/2512.14026/) (2025)
- [Brain-WM: Brain Glioblastoma World Model](https://mlsanity.com/paper/2603.07562/) (2026)
- [No Data? No Problem: Robust Vision-Tabular Learning with Missing Values](https://mlsanity.com/paper/2512.19602/) (2025)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [HyDA: Hypernetworks for Test Time Domain Adaptation in Medical Imaging Analysis](https://mlsanity.com/paper/2503.04979/) (2025)
- [Language-Enhanced Generative Modeling for Amyloid PET Synthesis from MRI and Blood Biomarkers](https://mlsanity.com/paper/2511.02206/) (2025)
- [LeMoF: Level-guided Multimodal Fusion for Heterogeneous Clinical Data](https://mlsanity.com/paper/2601.10092/) (2026)
- [AMF-MedIT: An Efficient Align-Modulation-Fusion Framework for Medical Image-Tabular Data](https://mlsanity.com/paper/2506.19439/) (2025)
- [Hypernetwork-Based Adaptive Aggregation for Multimodal Multiple-Instance Learning in Predicting Coronary Calcium Debulking](https://mlsanity.com/paper/2601.21479/) (2026)
- [MIRAGE: Knowledge Graph-Guided Cross-Cohort MRI Synthesis for Alzheimer's Disease Prediction](https://mlsanity.com/paper/2603.02434/) (2026)
- [Disentangled Fine-Grained Prototype Learning for Incomplete Image-Tabular Classification](https://mlsanity.com/paper/2606.05455/) (2026)
- [The Importance of Encoder Choice:A Tabular-Image Study](https://mlsanity.com/paper/2607.07756/) (2026)
- [Parameter-Dynamic Adaptive Fusion and Calibration Network for RGBT Tracking](https://mlsanity.com/paper/2608.01807/) (2026)
- [PANDA - Prototype-Anchored Alignment for Partially Unpaired Multimodal Learning, with Applications to Alzheimers MRI and TCGA Pathology](https://mlsanity.com/paper/2608.25970/) (2026)
- [MMAP: Multimodal Missing-Aware Pretraining for Longitudinal Alzheimer's Prediction](https://mlsanity.com/paper/2609.26617/) (2026)

## Abstract (arXiv)

The integration of diverse clinical modalities such as medical imaging and the tabular data extracted from patients' Electronic Health Records (EHRs) is a crucial aspect of modern healthcare. Integrative analysis of multiple sources can provide a comprehensive understanding of the clinical condition of a patient, improving diagnosis and treatment decision. Deep Neural Networks (DNNs) consistently demonstrate outstanding performance in a wide range of multimodal tasks in the medical domain. However, the complex endeavor of effectively merging medical imaging with clinical, demographic and genetic information represented as numerical tabular data remains a highly active and ongoing research pursuit. We present a novel framework based on hypernetworks to fuse clinical imaging and tabular data by conditioning the image processing on the EHR's values and measurements. This approach aims to leverage the complementary information present in these modalities to enhance the accuracy of various medical applications. We demonstrate the strength and generality of our method on two different brain Magnetic Resonance Imaging (MRI) analysis tasks, namely, brain age prediction conditioned by subject's sex and multi-class Alzheimer's Disease (AD) classification conditioned by tabular data. We show that our framework outperforms both single-modality models and state-of-the-art MRI tabular data fusion methods. A link to our code can be found at https://github.com/daniel4725/HyperFusion