# Exploring Multi-Modality Dynamics: Insights and Challenges in Multimodal Fusion for Biomedical Tasks

arXiv: https://arxiv.org/abs/2411.00725 · HTML: https://mlsanity.com/paper/2411.00725/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 1 Nov 2024.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [A Survey of Multimodal Information Fusion for Smart Healthcare: Mapping the Journey from Data to Wisdom](https://mlsanity.com/paper/2306.11963/) (2023)
- [Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis](https://mlsanity.com/paper/2206.06367/) (2022)
- [Dynamic Deep Multi-modal Fusion for Image Privacy Prediction](https://mlsanity.com/paper/1902.10796/) (2019)

## Abstract (arXiv)

This paper investigates the MM dynamics approach proposed by Han et al. (2022) for multi-modal fusion in biomedical classification tasks. The MM dynamics algorithm integrates feature-level and modality-level informativeness to dynamically fuse modalities for improved classification performance. However, our analysis reveals several limitations and challenges in replicating and extending the results of MM dynamics. We found that feature informativeness improves performance and explainability, while modality informativeness does not provide significant advantages and can lead to performance degradation. Based on these results, we have extended feature informativeness to image data, resulting in the development of Image MM dynamics. Although this approach showed promising qualitative results, it did not outperform baseline methods quantitatively.