mlsanityData to 14 Oct 2026

Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis

arXiv 1905.05812 ↗First posted 14 May 2019cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Related tasks often have inter-dependence on each other and perform better when solved in a joint framework. In this paper, we present a deep multi-task learning framework that jointly performs sentiment and emotion analysis both. The multi-modal inputs (i.e., text, acoustic and visual frames) of a video convey diverse and distinctive information, and usually do not have equal contribution in the decision making. We propose a context-level inter-modal attention framework for simultaneously predicting the sentiment and expressed emotions of an utterance. We evaluate our proposed approach on CMU-MOSEI dataset for multi-modal sentiment and emotion analysis. Evaluation results suggest that multi-task learning framework offers improvement over the single-task framework. The proposed approach reports new state-of-the-art performance for both sentiment analysis and emotion analysis.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 3; 2023: 8; 2024: 6; 2025: 5; 2026: 2.

39 papers.

Sort:

By area:

  1. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
  2. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  3. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
  4. 2022UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognitioncited by 61
  5. 2022Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representationscited by 48
  6. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  7. 2021Multimodal End-to-End Sparse Model for Emotion Recognitioncited by 40
  8. 2021Multimodal Learning using Optimal Transport for Sarcasm and Humor Detectioncited by 37
  9. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  10. 2021Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversationscited by 33
  11. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  12. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
27 more
  1. 2023Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognitioncited by 13
  2. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 13
  3. 2021Multi-Task Learning with Sentiment, Emotion, and Target Detection to Recognize Hate Speech and Offensive Languagecited by 12
  4. 2021ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptorscited by 12
  5. 2023Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotioncited by 12
  6. 2024FACTOID: FACtual enTailment fOr hallucInation Detectioncited by 10
  7. 2021Exercise? I thought you said 'Extra Fries': Leveraging Sentence Demarcations and Multi-hop Attention for Meme Affect Analysiscited by 9
  8. 2024UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Causecited by 8
  9. 2023A Comprehensive Survey on Affective Computing; Challenges, Trends, Applications, and Future Directionscited by 7
  10. 2024ASEM: Enhancing Empathy in Chatbot through Attention-based Sentiment and Emotion Modelingcited by 7
  11. 2021Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysiscited by 5
  12. 2021TI-Capsule: Capsule Network for Stock Exchange Predictioncited by 4
  13. 2021Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspectscited by 4
  14. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  15. 2021Weakly-supervised Multi-task Learning for Multimodal Affect Recognitioncited by 3
  16. 2023HausaNLP at SemEval-2023 Task 12: Leveraging African Low Resource TweetData for Sentiment Analysiscited by 3
  17. 2024Infer Induced Sentiment of Comment Response to Video: A New Task, Dataset and Baselinecited by 2
  18. 2025Multi-Modality Collaborative Learning for Sentiment Analysiscited by 1
  19. 2026Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathycited by 1
  20. 2023Core Challenges in Embodied Vision-Language Planningcited by 0
  21. 2023Emotion Embeddings unicodex2014 Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasetscited by 0
  22. 2023Self-MI: Efficient Multimodal Fusion via Self-Supervised Multi-Task Learning with Auxiliary Mutual Information Maximizationcited by 0
  23. 2025Hierarchical Adaptive Expert for Multimodal Sentiment Analysiscited by 0
  24. 2025Dynamic Domain Information Modulation Algorithm for Multi-domain Sentiment Analysiscited by 0
  25. 2025Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognitioncited by 0
  26. 2025Emotion-Enhanced Multi-Task Learning with LLMs for Aspect Category Sentiment Analysiscited by 0
  27. 2026CMHL: Contrastive Multi-Head Learning for Emotionally Consistent Text Classificationcited by 0