# Multimodal Fusion Learning with Dual Attention for Medical Imaging

arXiv: https://arxiv.org/abs/2412.01248 · HTML: https://mlsanity.com/paper/2412.01248/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 41 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Multimodal Multi-Head Convolutional Attention with Various Kernel Sizes for Medical Image Super-Resolution](https://mlsanity.com/paper/2204.04218/) (2022)
- [Is the aspect ratio of cells important in deep learning? A robust comparison of deep learning methods for multi-scale cytopathology cell image classification: from convolutional neural networks to visual transformers](https://mlsanity.com/paper/2105.07402/) (2021)
- [Data Augmentation for Skin Lesion using Self-Attention based Progressive Generative Adversarial Network](https://mlsanity.com/paper/1910.11960/) (2019)
- [A Multi-modal Fusion Framework Based on Multi-task Correlation Learning for Cancer Prognosis Prediction](https://mlsanity.com/paper/2201.10353/) (2022)
- [Assessing the Impact of Attention and Self-Attention Mechanisms on the Classification of Skin Lesions](https://mlsanity.com/paper/2112.12748/) (2021)
- [Cross-Task Attention Network: Improving Multi-Task Learning for Medical Imaging Applications](https://mlsanity.com/paper/2309.03837/) (2023)

## Built on by (most cited first, 5 of 5)

- [A Comprehensive Review of Techniques, Algorithms, Advancements, Challenges, and Clinical Applications of Multi-modal Medical Image Fusion for Improved Diagnosis](https://mlsanity.com/paper/2505.14715/) (2025)
- [Effective and Robust Multimodal Medical Image Analysis](https://mlsanity.com/paper/2602.15346/) (2026)
- [HyPCA-Net: Advancing Multimodal Fusion in Medical Image Analysis](https://mlsanity.com/paper/2602.16245/) (2026)
- [GLEAM: A Multimodal Imaging Dataset and HAMM for Glaucoma Classification](https://mlsanity.com/paper/2603.12800/) (2026)
- [Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention](https://mlsanity.com/paper/2607.19086/) (2026)

## Abstract (arXiv)

Multimodal fusion learning has shown significant promise in classifying various diseases such as skin cancer and brain tumors. However, existing methods face three key limitations. First, they often lack generalizability to other diagnosis tasks due to their focus on a particular disease. Second, they do not fully leverage multiple health records from diverse modalities to learn robust complementary information. And finally, they typically rely on a single attention mechanism, missing the benefits of multiple attention strategies within and across various modalities. To address these issues, this paper proposes a dual robust information fusion attention mechanism (DRIFA) that leverages two attention modules, i.e. multi-branch fusion attention module and the multimodal information fusion attention module. DRIFA can be integrated with any deep neural network, forming a multimodal fusion learning framework denoted as DRIFA-Net. We show that the multi-branch fusion attention of DRIFA learns enhanced representations for each modality, such as dermoscopy, pap smear, MRI, and CT-scan, whereas multimodal information fusion attention module learns more refined multimodal shared representations, improving the network's generalization across multiple tasks and enhancing overall performance. Additionally, to estimate the uncertainty of DRIFA-Net predictions, we have employed an ensemble Monte Carlo dropout strategy. Extensive experiments on five publicly available datasets with diverse modalities demonstrate that our approach consistently outperforms state-of-the-art methods. The code is available at https://github.com/misti1203/DRIFA-Net.