# Multimodal Multi-loss Fusion Network for Sentiment Analysis

arXiv: https://arxiv.org/abs/2308.00264 · HTML: https://mlsanity.com/paper/2308.00264/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Efficient Low-rank Multimodal Fusion with Modality-Specific Factors](https://mlsanity.com/paper/1806.00064/) (2018)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos](https://mlsanity.com/paper/1606.06259/) (2016)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis](https://mlsanity.com/paper/1911.05544/) (2019)
- [A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2006.15955/) (2020)
- [TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2009.02902/) (2020)
- [Multimodal End-to-End Sparse Model for Emotion Recognition](https://mlsanity.com/paper/2103.09666/) (2021)
- [Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation](https://mlsanity.com/paper/2002.08267/) (2020)
- [Sparse Spatial Transformers for Few-Shot Learning](https://mlsanity.com/paper/2109.12932/) (2021)
- [TEASEL: A Transformer-Based Speech-Prefixed Language Model](https://mlsanity.com/paper/2109.05522/) (2021)
- [Modality-based Factorization for Multimodal Fusion](https://mlsanity.com/paper/1811.12624/) (2018)

## Built on by (most cited first, 27 of 27)

- [Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances](https://mlsanity.com/paper/2407.21315/) (2024)
- [Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark](https://mlsanity.com/paper/2504.16427/) (2025)
- [Akan Cinematic Emotions (ACE): A Multimodal Multi-party Dataset for Emotion Recognition in Movie Dialogues](https://mlsanity.com/paper/2502.10973/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphs](https://mlsanity.com/paper/2505.01068/) (2025)
- [My Words Imply Your Opinion: Reader Agent-based Propagation Enhancement for Personalized Implicit Emotion Analysis](https://mlsanity.com/paper/2412.07367/) (2024)
- [Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approach](https://mlsanity.com/paper/2505.07902/) (2025)
- [Learning More with Less: Self-Supervised Approaches for Low-Resource Speech Emotion Recognition](https://mlsanity.com/paper/2506.02059/) (2025)
- [EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Experts](https://mlsanity.com/paper/2604.12579/) (2026)
- [Enhancing Multimodal Affective Analysis with Learned Live Comment Features](https://mlsanity.com/paper/2410.16407/) (2024)
- [RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer](https://mlsanity.com/paper/2506.11465/) (2025)
- [Structures Meet Semantics: Multimodal Fusion via Graph Contrastive Learning](https://mlsanity.com/paper/2508.18322/) (2025)
- [National Institute on Aging PREPARE Challenge: Early Detection of Cognitive Impairment Using Speech -- The SpeechCARE Solution](https://mlsanity.com/paper/2511.08132/) (2025)
- [GSIFN: A Graph-Structured and Interlaced-Masked Multimodal Transformer-based Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2408.14809/) (2024)
- [MEANT: Multimodal Encoder for Antecedent Information](https://mlsanity.com/paper/2411.06616/) (2024)
- [SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment](https://mlsanity.com/paper/2411.18162/) (2024)
- [Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation](https://mlsanity.com/paper/2504.13465/) (2025)
- [Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2505.14143/) (2025)
- [Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarization](https://mlsanity.com/paper/2507.19356/) (2025)
- [MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2508.09145/) (2025)
- [QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2510.27091/) (2025)
- [To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition](https://mlsanity.com/paper/2605.04877/) (2026)
- [GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attention](https://mlsanity.com/paper/2606.06249/) (2026)
- [Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysis](https://mlsanity.com/paper/2608.16201/) (2026)
- [The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methods](https://mlsanity.com/paper/2609.11247/) (2026)
- [Empath: Tracing Multi-Level Emotion Dynamics in Crisis Counseling Dialogues](https://mlsanity.com/paper/2609.29056/) (2026)
- [InstMoE: Adaptive Multimodal Routing with Specialized Experts](https://mlsanity.com/paper/2610.05111/) (2026)

## Abstract (arXiv)

This paper investigates the optimal selection and fusion of feature encoders across multiple modalities and combines these in one neural network to improve sentiment detection. We compare different fusion methods and examine the impact of multi-loss training within the multi-modality fusion network, identifying surprisingly important findings relating to subnet performance. We have also found that integrating context significantly enhances model performance. Our best model achieves state-of-the-art performance for three datasets (CMU-MOSI, CMU-MOSEI and CH-SIMS). These results suggest a roadmap toward an optimized feature selection and fusion approach for enhancing sentiment detection in neural networks.