mlsanityData to 7 Oct 2026

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition

arXiv 2211.11256 ↗First posted 21 Nov 2022cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Multimodal sentiment analysis (MSA) and emotion recognition in conversation (ERC) are key research topics for computers to understand human behaviors. From a psychological perspective, emotions are the expression of affect or feelings during a short period, while sentiments are formed and held for a longer period. However, most existing works study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. In this paper, we propose a multimodal sentiment knowledge-sharing framework (UniMSE) that unifies MSA and ERC tasks from features, labels, and models. We perform modality fusion at the syntactic and semantic levels and introduce contrastive learning between modalities and samples to better capture the difference and consistency between sentiments and emotions. Experiments on four public benchmark datasets, MOSI, MOSEI, MELD, and IEMOCAP, demonstrate the effectiveness of the proposed method and achieve consistent improvements compared with state-of-the-art methods.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 53 of its 91 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

53 of 91 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016Layer Normalizationcited by 5,579
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2020Supervised Contrastive Learningcited by 2,803
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  12. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
38 more
  1. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  2. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  3. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  4. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  5. 2019DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversationcited by 184
  6. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
  7. 2018Learning Factorized Multimodal Representationscited by 171
  8. 2021A Unified Generative Framework for Various NER Subtaskscited by 141
  9. 2020COSMIC: COmmonSense knowledge for eMotion Identification in Conversationscited by 135
  10. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  11. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  12. 2021MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversationcited by 99
  13. 2021Directed Acyclic Graph Network for Conversational Emotion Recognitioncited by 97
  14. 2021A Unified Generative Framework for Aspect-Based Sentiment Analysiscited by 97
  15. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  16. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  17. 2020Discovering New Intents with Deep Aligned Clusteringcited by 68
  18. 2019Deep Unknown Intent Detection with Margin Losscited by 66
  19. 2019Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinementcited by 56
  20. 2020Deep Open Intent Classification with Adaptive Decision Boundarycited by 56
  21. 2021Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detectioncited by 56
  22. 2022MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversationscited by 52
  23. 2019Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusioncited by 45
  24. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  25. 2019Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysiscited by 39
  26. 2021UniRE: A Unified Label Space for Entity Relation Extractioncited by 38
  27. 2021Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialog State Trackingcited by 35
  28. 2021CoMPM: Context Modeling with Speaker's Pre-trained Memory Tracking for Emotion Recognition in Conversationcited by 35
  29. 2021Contrast and Generation Make BART a Good Dialogue Emotion Recognizercited by 35
  30. 2022SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generationcited by 34
  31. 2022COGMEN: COntextualized GNN based Multimodal Emotion recognitioNcited by 29
  32. 2022Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systemscited by 28
  33. 2022SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understandingcited by 28
  34. 2021UniMS: A Unified Framework for Multimodal Summarization with Knowledge Distillationcited by 23
  35. 2022M-SENA: An Integrated Platform for Multimodal Sentiment Analysiscited by 21
  36. 2022A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slotscited by 13
  37. 2021ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptorscited by 12
  38. 2020A Post-processing Method for Detecting Unknown Intent of Dialogue System via Pre-trained Deep Neural Network Classifiercited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 21; 2025: 12; 2026: 13.

61 papers.

Sort:

By area:

  1. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instructcited by 42
  2. 2023DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversationscited by 29
  3. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  4. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 20
  5. 2024Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representationscited by 18
  6. 2024Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learningcited by 18
  7. 2023UniSA: Unified Generative Framework for Sentiment Analysiscited by 17
  8. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functionscited by 16
  9. 2023CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videoscited by 14
  10. 2023CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognitioncited by 14
  11. 2023Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignmentcited by 13
  12. 2023Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognitioncited by 13
38 more
  1. 2024PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysiscited by 13
  2. 2024Recent Advances in Multimodal Affective Computing: An NLP Perspectivecited by 12
  3. 2023Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance Representationcited by 11
  4. 2024BiosERC: Integrating Biography Speakers Supported by LLMs for ERC Taskscited by 11
  5. 2023Empathetic Response Generation via Emotion Cause Transition Graphcited by 8
  6. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
  7. 2023VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learningcited by 6
  8. 2026Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysiscited by 6
  9. 2023Self-Explanation Prompting Improves Dialogue Understanding in Large Language Modelscited by 5
  10. 2023Emotion Prediction Oriented method with Multiple Supervisions for Emotion-Cause Pair Extractioncited by 4
  11. 2024Triple Disentangled Representation Learning for Multimodal Affective Analysiscited by 4
  12. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  13. 2024RCLMuFN: Relational Context Learning and Multiplex Fusion Network for Multimodal Sarcasm Detectioncited by 4
  14. 2026CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learningcited by 4
  15. 2023SeqAug: Sequential Feature Resampling as a modality agnostic augmentation methodcited by 3
  16. 2023Hierarchical Audio-Visual Information Fusion with Multi-label Joint Decoding for MER 2023cited by 3
  17. 2024EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understandingcited by 3
  18. 2024Meta-Learn Unimodal Signals with Weak Supervision for Multimodal Sentiment Analysiscited by 2
  19. 2024Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attentioncited by 2
  20. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
  21. 2023Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systemscited by 1
  22. 2024AMuSE: Adaptive Multimodal Analysis for Speaker Emotion Recognition in Group Conversationscited by 1
  23. 2024Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Bettercited by 1
  24. 2024Multimodal Sentiment Analysis Based on Causal Reasoningcited by 1
  25. 2025A Heterogeneous Multimodal Graph Learning Framework for Recognizing User Emotions in Social Networkscited by 1
  26. 2025Omni-SILA: Towards Omni-scene Driven Visual Sentiment Identifying, Locating and Attributing in Videoscited by 1
  27. 2025DashFusion: Dual-stream Alignment with Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysiscited by 1
  28. 2026Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflectioncited by 1
  29. 2026ModalImmune: Immunity Driven Unlearning via Self Destructive Trainingcited by 1
  30. 2026A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversationscited by 1
  31. 2024AM^2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learningcited by 0
  32. 2024CARBD-Ko: A Contextually Annotated Review Benchmark Dataset for Aspect-Level Sentiment Classification in Koreancited by 0
  33. 2024WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognitioncited by 0
  34. 2024Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognitioncited by 0
  35. 2025Hybrid Emotion Recognition: Enhancing Customer Interactions Through Acoustic and Textual Analysiscited by 0
  36. 2025Hierarchical Adaptive Expert for Multimodal Sentiment Analysiscited by 0
  37. 2025Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognitioncited by 0
  38. 2025QiNN-QJ: A Quantum-inspired Neural Network with Quantum Jump for Multimodal Sentiment Analysiscited by 0

50 most cited of 61 All 61 (JSON)