# Learn to Combine Modalities in Multimodal Deep Learning

arXiv: https://arxiv.org/abs/1805.11730 · HTML: https://mlsanity.com/paper/1805.11730/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2018.

## Built on by (most cited first, 29 of 29)

- [Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)](https://mlsanity.com/paper/2203.12221/) (2022)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Dynamic Multimodal Fusion](https://mlsanity.com/paper/2204.00102/) (2022)
- [FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context](https://mlsanity.com/paper/2203.02113/) (2022)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Graph Neural Networks in IoT: A Survey](https://mlsanity.com/paper/2203.15935/) (2022)
- [A Survey on Multi-modal Summarization](https://mlsanity.com/paper/2109.05199/) (2021)
- [UniCat: Crafting a Stronger Fusion Baseline for Multimodal Re-Identification](https://mlsanity.com/paper/2310.18812/) (2023)
- [Revisiting Ensembling in One-Shot Federated Learning](https://mlsanity.com/paper/2411.07182/) (2024)
- [A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition](https://mlsanity.com/paper/2111.02172/) (2021)
- [Class Based Thresholding in Early Exit Semantic Segmentation Networks](https://mlsanity.com/paper/2210.15621/) (2022)
- [Employing Multimodal Machine Learning for Stress Detection](https://mlsanity.com/paper/2306.09385/) (2023)
- [Integrating Chemical Language and Molecular Graph in Multimodal Fused Deep Learning for Drug Property Prediction](https://mlsanity.com/paper/2312.17495/) (2023)
- [On Robustness in Multimodal Learning](https://mlsanity.com/paper/2304.04385/) (2023)
- [TACFN: Transformer-based Adaptive Cross-modal Fusion Network for Multimodal Emotion Recognition](https://mlsanity.com/paper/2505.06536/) (2025)
- [Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems](https://mlsanity.com/paper/2111.02064/) (2021)
- [Shape-Net: Room Layout Estimation from Panoramic Images Robust to Occlusion using Knowledge Distillation with 3D Shapes as Additional Inputs](https://mlsanity.com/paper/2304.12624/) (2023)
- [Data Fusion for Deep Learning on Transport Mode Detection: A Case Study](https://mlsanity.com/paper/2106.05876/) (2021)
- [Interpretable Prediction of Pulmonary Hypertension in Newborns using Echocardiograms](https://mlsanity.com/paper/2203.13038/) (2022)
- [Classification of Quasars, Galaxies, and Stars in the Mapping of the Universe Multi-modal Deep Learning](https://mlsanity.com/paper/2205.10745/) (2022)
- [3D object quality prediction for Metal Jet Printer with Multimodal thermal encoder](https://mlsanity.com/paper/2404.11776/) (2024)
- [Camera Model Identification Using Audio and Visual Content from Videos](https://mlsanity.com/paper/2406.17916/) (2024)
- [Deep Metric Loss for Multimodal Learning](https://mlsanity.com/paper/2308.10486/) (2023)
- [Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach](https://mlsanity.com/paper/2406.00901/) (2024)
- [Sequence-to-Sequence Multi-Modal Speech In-Painting](https://mlsanity.com/paper/2406.01321/) (2024)
- [Multi-Task Adversarial Variational Autoencoder for Estimating Biological Brain Age with Multimodal Neuroimaging](https://mlsanity.com/paper/2411.10100/) (2024)
- [Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure](https://mlsanity.com/paper/2504.10049/) (2025)
- [Retrieval-Augmented VLMs for Multimodal Melanoma Diagnosis](https://mlsanity.com/paper/2509.08338/) (2025)

## Abstract (arXiv)

Combining complementary information from multiple modalities is intuitively appealing for improving the performance of learning-based approaches. However, it is challenging to fully leverage different modalities due to practical challenges such as varying levels of noise and conflicts between modalities. Existing methods do not adopt a joint approach to capturing synergies between the modalities while simultaneously filtering noise and resolving conflicts on a per sample basis. In this work we propose a novel deep neural network based technique that multiplicatively combines information from different source modalities. Thus the model training process automatically focuses on information from more reliable modalities while reducing emphasis on the less reliable modalities. Furthermore, we propose an extension that multiplicatively combines not only the single-source modalities, but a set of mixtured source modalities to better capture cross-modal signal correlations. We demonstrate the effectiveness of our proposed technique by presenting empirical results on three multimodal classification tasks from different domains. The results show consistent accuracy improvements on all three tasks.