mlsanityData to 14 Oct 2026

Multimodal Multi-loss Fusion Network for Sentiment Analysis

arXiv 2308.00264 ↗First posted 1 Aug 2023cs.CL cs.AI cs.LGcited by 27

Not on the map. Why?

Abstract (arXiv)

This paper investigates the optimal selection and fusion of feature encoders across multiple modalities and combines these in one neural network to improve sentiment detection. We compare different fusion methods and examine the impact of multi-loss training within the multi-modality fusion network, identifying surprisingly important findings relating to subnet performance. We have also found that integrating context significantly enhances model performance. Our best model achieves state-of-the-art performance for three datasets (CMU-MOSI, CMU-MOSEI and CH-SIMS). These results suggest a roadmap toward an optimized feature selection and fusion approach for enhancing sentiment detection in neural networks.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 1 Aug 2023.

Builds on

20 of 32 references.

References · 20

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  4. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  5. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  6. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 433
  7. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 270
  8. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
  9. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 194
  10. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
  11. 2018Learning Factorized Multimodal Representationscited by 171
  12. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
8 more
  1. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  2. 2020A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysiscited by 46
  3. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  4. 2021Multimodal End-to-End Sparse Model for Emotion Recognitioncited by 40
  5. 2020Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversationcited by 27
  6. 2021Sparse Spatial Transformers for Few-Shot Learningcited by 14
  7. 2021TEASEL: A Transformer-Based Speech-Prefixed Language Modelcited by 8
  8. 2018Modality-based Factorization for Multimodal Fusioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 7.

27 papers.

Sort:

By area:

  1. 2024Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuancescited by 27
  2. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 9
  3. 2025Akan Cinematic Emotions (ACE): A Multimodal Multi-party Dataset for Emotion Recognition in Movie Dialoguescited by 6
  4. 2025A Review of Incorporating Psychological Theories in LLMscited by 6
  5. 2025Multimodal Transformers are Hierarchical Modal-wise Heterogeneous Graphscited by 5
  6. 2024My Words Imply Your Opinion: Reader Agent-based Propagation Enhancement for Personalized Implicit Emotion Analysiscited by 4
  7. 2025Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approachcited by 2
  8. 2025Learning More with Less: Self-Supervised Approaches for Low-Resource Speech Emotion Recognitioncited by 2
  9. 2026EEG-Based Multimodal Learning via Hyperbolic Mixture-of-Curvature Expertscited by 2
  10. 2024Enhancing Multimodal Affective Analysis with Learned Live Comment Featurescited by 1
  11. 2025RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformercited by 1
  12. 2025Structures Meet Semantics: Multimodal Fusion via Graph Contrastive Learningcited by 1
15 more
  1. 2025National Institute on Aging PREPARE Challenge: Early Detection of Cognitive Impairment Using Speech -- The SpeechCARE Solutioncited by 1
  2. 2024GSIFN: A Graph-Structured and Interlaced-Masked Multimodal Transformer-based Fusion Network for Multimodal Sentiment Analysiscited by 0
  3. 2024MEANT: Multimodal Encoder for Antecedent Informationcited by 0
  4. 2024SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environmentcited by 0
  5. 2025Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimationcited by 0
  6. 2025Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognitioncited by 0
  7. 2025Enhancing Speech Emotion Recognition Leveraging Aligning Timestamps of ASR Transcripts and Speaker Diarizationcited by 0
  8. 2025MoLAN: A Unified Modality-Aware Noise Dynamic Editing Framework for Multimodal Sentiment Analysiscited by 0
  9. 2025QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysiscited by 0
  10. 2026To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognitioncited by 0
  11. 2026GRAMformer: Any-Order Modality Interactions via Volumetric Multimodal Cross-Attentioncited by 0
  12. 2026Multi-Granularity Sentiment Integration for LLM-Based Multimodal Sentiment Analysiscited by 0
  13. 2026The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methodscited by 0
  14. 2026Empath: Tracing Multi-Level Emotion Dynamics in Crisis Counseling Dialoguescited by 0
  15. 2026InstMoE: Adaptive Multimodal Routing with Specialized Expertscited by 0