mlsanityData to 7 Oct 2026

A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning

arXiv 2312.05735 ↗First posted 10 Dec 2023cs.AIcited by 32

Not on the map. Why?

Abstract (arXiv)

Multi-modal conversation emotion recognition (MCER) aims to recognize and track the speaker's emotional state using text, speech, and visual information in the conversation scene. Analyzing and studying MCER issues is significant to affective computing, intelligent recommendations, and human-computer interaction fields. Unlike the traditional single-utterance multi-modal emotion recognition or single-modal conversation emotion recognition, MCER is a more challenging problem that needs to deal with more complex emotional interaction relationships. The critical issue is learning consistency and complementary semantics for multi-modal feature fusion based on emotional interaction relationships. To solve this problem, people have conducted extensive research on MCER based on deep learning technology, but there is still a lack of systematic review of the modeling methods. Therefore, a timely and comprehensive overview of MCER's recent advances in deep learning is of great significance to academia and industry. In this survey, we provide a comprehensive overview of MCER modeling methods and roughly divide MCER methods into four categories, i.e., context-free modeling, sequential context modeling, speaker-differentiated modeling, and speaker-relationship modeling. In addition, we further discuss MCER's publicly available popular datasets, multi-modal feature extraction methods, application areas, existing challenges, and future development directions. We hope that our review can help MCER researchers understand the current research status in emotion recognition, provide some inspiration, and develop more efficient models.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 50 of its 161 references that are in the map. First posted on arXiv 10 Dec 2023.

Builds on

50 of 161 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  5. 2019CNN-generated images are surprisingly easy to spot... for nowcited by 842
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  7. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  8. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  9. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  10. 2021Transformer in Transformercited by 632
  11. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  12. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
38 more
  1. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  2. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  3. 2018DialogueRNN: An Attentive RNN for Emotion Detection in Conversationscited by 216
  4. 2019DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversationcited by 184
  5. 2018Memory Fusion Network for Multi-view Sequential Learningcited by 167
  6. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  7. 2020COSMIC: COmmonSense knowledge for eMotion Identification in Conversationscited by 135
  8. 2017Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networkscited by 102
  9. 2021Directed Acyclic Graph Network for Conversational Emotion Recognitioncited by 97
  10. 2019Knowledge-Enriched Transformer for Emotion Detection in Textual Conversationscited by 92
  11. 2018Multi-attention Recurrent Network for Human Communication Comprehensioncited by 89
  12. 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
  13. 2022GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversationcited by 86
  14. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  15. 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
  16. 2021DialogueCRN: Contextual Reasoning Networks for Emotion Recognition in Conversationscited by 71
  17. 2021Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detectioncited by 56
  18. 2018CNN-Cert: An Efficient Framework for Certifying Robustness of Convolutional Neural Networkscited by 50
  19. 2022EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognitioncited by 48
  20. 2023CoCo: A Coupled Contrastive Framework for Unsupervised Domain Adaptive Graph Classificationcited by 46
  21. 2019Exploiting multi-CNN features in CNN-RNN based Dimensional Emotion Recognition on the OMG in-the-wild Datasetcited by 39
  22. 2016Select-Additive Learning: Improving Generalization in Multimodal Sentiment Analysiscited by 29
  23. 2020BiERU: Bidirectional Emotional Recurrent Unit for Conversational Sentiment Analysiscited by 28
  24. 2021The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stresscited by 27
  25. 2023Graph Information Bottleneck for Remote Sensing Segmentationcited by 24
  26. 2022GraphCFC: A Directed Graph Based Cross-Modal Feature Complementation Approach for Multimodal Conversational Emotion Recognitioncited by 20
  27. 2021EmoNet: A Transfer Learning Framework for Multi-Corpus Speech Emotion Recognitioncited by 17
  28. 2018Curriculum Learning for Speech Emotion Recognition from Crowdsourced Labelscited by 16
  29. 2019Multi-Task Semi-Supervised Adversarial Autoencoding for Speech Emotion Recognitioncited by 15
  30. 2017Benchmarking Multimodal Sentiment Analysiscited by 11
  31. 2022M2R2: Missing-Modality Robust emotion Recognition framework with iterative data augmentationcited by 8
  32. 2022Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errorscited by 7
  33. 2020SemGloVe: Semantic Co-occurrences for GloVe from BERTcited by 6
  34. 2023Relevance Feedback with Brain Signalscited by 4
  35. 2017Advanced LSTM: A Study about Better Time Dependency Modeling in Emotion Recognitioncited by 3
  36. 2018Integrating Recurrence Dynamics for Speech Emotion Recognitioncited by 3
  37. 2021Privacy-Protecting Techniques for Behavioral Biometric Data: A Surveycited by 3
  38. 2022The Unboxing Experience: Exploration and Design of Initial Interactions Between Children and Social Robotscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 15; 2025: 5; 2026: 10.

32 papers.

Sort:

By area:

  1. 2024Revisiting Multi-modal Emotion Learning with Broad State Space Models and Probability-guidance Fusioncited by 33
  2. 2024Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrumcited by 33
  3. 2023DER-GCN: Dialogue and Event Relation-Aware Graph Convolutional Neural Network for Multimodal Dialogue Emotion Recognitioncited by 23
  4. 2024Contrastive Graph Representation Learning with Adversarial Cross-view Reconstruction and Information Bottleneckcited by 21
  5. 2024SpeGCL: Self-supervised Graph Spectrum Contrastive Learning without Positive Samplescited by 20
  6. 2024Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Predictioncited by 20
  7. 2024Masked Graph Learning with Recurrent Alignment for Multimodal Emotion Recognition in Conversationcited by 19
  8. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Surveycited by 19
  9. 2023Adversarial Representation with Intra-Modal and Inter-Modal Graph Contrastive Learning for Multimodal Emotion Recognitioncited by 12
  10. 2024MCSFF: Multi-modal Consistency and Specificity Fusion Framework for Entity Alignmentcited by 8
  11. 2024Graph Contrastive Learning via Cluster-refined Negative Sampling for Semi-supervised Text Classificationcited by 8
  12. 2024A Multi-view Mask Contrastive Learning Graph Convolutional Neural Network for Age Estimationcited by 7
20 more
  1. 2026The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detectioncited by 6
  2. 2024Contrastive Multi-graph Learning with Neighbor Hierarchical Sifting for Semi-supervised Text Classificationcited by 5
  3. 2024SEG:Seeds-Enhanced Iterative Refinement Graph Neural Network for Entity Alignmentcited by 4
  4. 2024GroupFace: Imbalanced Age Estimation Based on Multi-hop Attention Graph Convolutional Network and Group-aware Margin Optimizationcited by 3
  5. 2026Dual-branch Graph Domain Adaptation for Cross-scenario Multi-modal Emotion Recognitioncited by 2
  6. 2024SE-GCL: An Event-Based Simple and Effective Graph Contrastive Learning for Text Representationcited by 1
  7. 2025LRA-GNN: Latent Relation-Aware Graph Neural Network with Initial and Dynamic Residual for Facial Age Estimationcited by 1
  8. 2025A Flow Model with Low-Rank Transformers for Incomplete Multimodal Survival Analysiscited by 1
  9. 2026Memory Bear AI Memory Science Engine for Multimodal Affective Intelligence: A Technical Reportcited by 1
  10. 2024An Effective Index for Truss-based Community Search on Large Directed Graphscited by 0
  11. 2024Utilizing Graph Generation for Enhanced Domain Adaptive Object Detectioncited by 0
  12. 2025SE-GNN: Seed Expanded-Aware Graph Neural Network with Iterative Optimization for Semi-supervised Entity Alignmentcited by 0
  13. 2025GSDNet: Revisiting Incomplete Multimodal-Diffusion from Graph Spectrum Perspective for Conversation Emotion Recognitioncited by 0
  14. 2026TimeGNN-Augmented Hybrid-Action MARL for Fine-Grained Task Partitioning and Energy-Aware Offloading in MECcited by 0
  15. 2026AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognitioncited by 0
  16. 2026Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversationscited by 0
  17. 2026Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognitioncited by 0
  18. 2026Disentangled Dual-Branch Graph Learning for Conversational Emotion Recognitioncited by 0
  19. 2026To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognitioncited by 0
  20. 2026C2MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learningcited by 0