# Dense Multimodal Fusion for Hierarchically Joint Representation

arXiv: https://arxiv.org/abs/1810.03414 · HTML: https://mlsanity.com/paper/1810.03414/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2018.

## Built on by (most cited first, 8 of 8)

- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [ImageArg: A Multi-modal Tweet Dataset for Image Persuasiveness Mining](https://mlsanity.com/paper/2209.06416/) (2022)
- [Improved Multimodal Fusion for Small Datasets with Auxiliary Supervision](https://mlsanity.com/paper/2304.00379/) (2023)
- [Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning](https://mlsanity.com/paper/2405.02766/) (2024)
- [Overview of ImageArg-2023: The First Shared Task in Multimodal Argument Mining](https://mlsanity.com/paper/2310.12172/) (2023)
- [Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression](https://mlsanity.com/paper/2208.00919/) (2022)
- [FROG: Effective Friend Recommendation in Online Games via Modality-aware User Preferences](https://mlsanity.com/paper/2504.09428/) (2025)

## Abstract (arXiv)

Multiple modalities can provide more valuable information than single one by describing the same contents in various ways. Hence, it is highly expected to learn effective joint representation by fusing the features of different modalities. However, previous methods mainly focus on fusing the shallow features or high-level representations generated by unimodal deep networks, which only capture part of the hierarchical correlations across modalities. In this paper, we propose to densely integrate the representations by greedily stacking multiple shared layers between different modality-specific networks, which is named as Dense Multimodal Fusion (DMF). The joint representations in different shared layers can capture the correlations in different levels, and the connection between shared layers also provides an efficient way to learn the dependence among hierarchical correlations. These two properties jointly contribute to the multiple learning paths in DMF, which results in faster convergence, lower training loss, and better performance. We evaluate our model on three typical multimodal learning tasks, including audiovisual speech recognition, cross-modal retrieval, and multimodal classification. The noticeable performance in the experiments demonstrates that our model can learn more effective joint representation.