# Modality-based Factorization for Multimodal Fusion

arXiv: https://arxiv.org/abs/1811.12624 · HTML: https://mlsanity.com/paper/1811.12624/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2018.

## Built on by (most cited first, 7 of 7)

- [Multimodal Multi-loss Fusion Network for Sentiment Analysis](https://mlsanity.com/paper/2308.00264/) (2023)
- [Hybrid Multimodal Fusion for Humor Detection](https://mlsanity.com/paper/2209.11949/) (2022)
- [TextMI: Textualize Multimodal Information for Integrating Non-verbal Cues in Pre-trained Language Models](https://mlsanity.com/paper/2303.15430/) (2023)
- [Quantum Cognitively Motivated Decision Fusion for Video Sentiment Analysis](https://mlsanity.com/paper/2101.04406/) (2021)
- [CT2C-QA: Multimodal Question Answering over Chinese Text, Table and Chart](https://mlsanity.com/paper/2410.21414/) (2024)
- [FOAA: Flattened Outer Arithmetic Attention For Multimodal Tumor Classification](https://mlsanity.com/paper/2403.06339/) (2024)
- [Cooperative Sentiment Agents for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2404.12642/) (2024)

## Abstract (arXiv)

We propose a novel method, Modality-based Redundancy Reduction Fusion (MRRF), for understanding and modulating the relative contribution of each modality in multimodal inference tasks. This is achieved by obtaining an (M+1)-way tensor to consider the high-order relationships between M modalities and the output layer of a neural network model. Applying a modality-based tensor factorization method, which adopts different factors for different modalities, results in removing information present in a modality that can be compensated by other modalities, with respect to model outputs. This helps to understand the relative utility of information in each modality. In addition it leads to a less complicated model with less parameters and therefore could be applied as a regularizer avoiding overfitting. We have applied this method to three different multimodal datasets in sentiment analysis, personality trait recognition, and emotion recognition. We are able to recognize relationships and relative importance of different modalities in these tasks and achieves a 1% to 4% improvement on several evaluation measures compared to the state-of-the-art for all three tasks.