# Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects

arXiv: https://arxiv.org/abs/2505.20511 · HTML: https://mlsanity.com/paper/2505.20511/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 123 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](https://mlsanity.com/paper/2010.02795/) (2020)
- [FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective](https://mlsanity.com/paper/2311.06190/) (2023)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networks](https://mlsanity.com/paper/1708.04299/) (2017)
- [Hierarchical Gating Networks for Sequential Recommendation](https://mlsanity.com/paper/1906.09217/) (2019)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [EmotionLines: An Emotion Corpus of Multi-Party Conversations](https://mlsanity.com/paper/1802.08379/) (2018)
- [Multimodal Alignment and Fusion: A Survey](https://mlsanity.com/paper/2411.17040/) (2024)
- [M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database](https://mlsanity.com/paper/2205.10237/) (2022)
- [Multimodal Fusion on Low-quality Data: A Comprehensive Survey](https://mlsanity.com/paper/2404.18947/) (2024)
- [EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition](https://mlsanity.com/paper/2203.13504/) (2022)
- [Survey on Emotional Body Gesture Recognition](https://mlsanity.com/paper/1801.07481/) (2018)
- [M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2206.02187/) (2022)
- [Hierarchical Self-supervised Augmented Knowledge Distillation](https://mlsanity.com/paper/2107.13715/) (2021)
- [Revisiting Graph Contrastive Learning from the Perspective of Graph Spectrum](https://mlsanity.com/paper/2210.02330/) (2022)
- [Real-Time Emotion Recognition via Attention Gated Hierarchical Memory Network](https://mlsanity.com/paper/1911.09075/) (2019)
- [Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum](https://mlsanity.com/paper/2404.17862/) (2024)

## Built on by (most cited first, 11 of 11)

- [Learning More with Less: Self-Supervised Approaches for Low-Resource Speech Emotion Recognition](https://mlsanity.com/paper/2506.02059/) (2025)
- [SCoPE: Shift-Aware Speaker-Conditioned Priors for Emotion Recognition in Conversations](https://mlsanity.com/paper/2607.20445/) (2026)
- [Decoding Student Minds: Leveraging Conversational Agents for Psychological and Learning Analysis](https://mlsanity.com/paper/2512.10441/) (2025)
- [AMB-DSGDN: Adaptive Modality-Balanced Dynamic Semantic Graph Differential Network for Multimodal Emotion Recognition](https://mlsanity.com/paper/2603.10043/) (2026)
- [CMHL: Contrastive Multi-Head Learning for Emotionally Consistent Text Classification](https://mlsanity.com/paper/2603.14078/) (2026)
- [SURE: Synergistic Uncertainty-aware Reasoning for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2604.01916/) (2026)
- [MISID: A Multimodal Multi-turn Dataset for Complex Intent Recognition in Strategic Deception Games](https://mlsanity.com/paper/2604.12700/) (2026)
- [AffectSeek: Agentic Affective Understanding in Long Videos under Vague User Queries](https://mlsanity.com/paper/2605.05640/) (2026)
- [MA-DLE: Speech-based Automatic Depression Level Estimation via Memory Augmentation](https://mlsanity.com/paper/2606.11197/) (2026)
- [Empath: Tracing Multi-Level Emotion Dynamics in Crisis Counseling Dialogues](https://mlsanity.com/paper/2609.29056/) (2026)
- [GeoMER: A Joint Euclidean-Hyperbolic Geometry Framework for Multimodal Emotion Recognition](https://mlsanity.com/paper/2610.18587/) (2026)

## Abstract (arXiv)

While text-based emotion recognition methods have achieved notable success, real-world dialogue systems often demand a more nuanced emotional understanding than any single modality can offer. Multimodal Emotion Recognition in Conversations (MERC) has thus emerged as a crucial direction for enhancing the naturalness and emotional understanding of human-computer interaction. Its goal is to accurately recognize emotions by integrating information from various modalities such as text, speech, and visual signals. This survey offers a systematic overview of MERC, including its motivations, core tasks, representative methods, and evaluation strategies. We further examine recent trends, highlight key challenges, and outline future directions. As interest in emotionally intelligent systems grows, this survey provides timely guidance for advancing MERC research.