# Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision

arXiv: https://arxiv.org/abs/2304.00379 · HTML: https://mlsanity.com/paper/2304.00379/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 29 references that are in the map.
- First posted on arXiv 1 Apr 2023.

## Builds on (most cited first, 4 of 4)

- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data](https://mlsanity.com/paper/2107.00648/) (2021)
- [Dense Multimodal Fusion for Hierarchically Joint Representation](https://mlsanity.com/paper/1810.03414/) (2018)

## Built on by (most cited first, 6 of 6)

- [HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data](https://mlsanity.com/paper/2311.09115/) (2023)
- [Conflict-Aware Multimodal Fusion for Ambivalence and Hesitancy Recognition](https://mlsanity.com/paper/2603.15818/) (2026)
- [DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era](https://mlsanity.com/paper/2404.12278/) (2024)
- [MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models](https://mlsanity.com/paper/2407.12309/) (2024)
- [TNF: Tri-branch Neural Fusion for Multimodal Medical Data Classification](https://mlsanity.com/paper/2403.01802/) (2024)
- [Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition](https://mlsanity.com/paper/2609.19148/) (2026)

## Abstract (arXiv)

Prostate cancer is one of the leading causes of cancer-related death in men worldwide. Like many cancers, diagnosis involves expert integration of heterogeneous patient information such as imaging, clinical risk factors, and more. For this reason, there have been many recent efforts toward deep multimodal fusion of image and non-image data for clinical decision tasks. Many of these studies propose methods to fuse learned features from each patient modality, providing significant downstream improvements with techniques like cross-modal attention gating, Kronecker product fusion, orthogonality regularization, and more. While these enhanced fusion operations can improve upon feature concatenation, they often come with an extremely high learning capacity, meaning they are likely to overfit when applied even to small or low-dimensional datasets. Rather than designing a highly expressive fusion operation, we propose three simple methods for improved multimodal fusion with small datasets that aid optimization by generating auxiliary sources of supervision during training: extra supervision, clinical prediction, and dense fusion. We validate the proposed approaches on prostate cancer diagnosis from paired histopathology imaging and tabular clinical features. The proposed methods are straightforward to implement and can be applied to any classification task with paired image and non-image data.