# M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database

arXiv: https://arxiv.org/abs/2205.10237 · HTML: https://mlsanity.com/paper/2205.10237/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 9 May 2022.

## Builds on (most cited first, 16 of 16)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advances](https://mlsanity.com/paper/1905.02947/) (2019)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [Emotion Detection on TV Show Transcripts with Sequence-based Convolutional Neural Networks](https://mlsanity.com/paper/1708.04299/) (2017)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [EmotionLines: An Emotion Corpus of Multi-Party Conversations](https://mlsanity.com/paper/1802.08379/) (2018)
- [DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition](https://mlsanity.com/paper/2012.08695/) (2020)
- [A Hierarchical Transformer with Speaker Modeling for Emotion Recognition in Conversation](https://mlsanity.com/paper/2012.14781/) (2020)

## Built on by (most cited first, 30 of 50)

- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens](https://mlsanity.com/paper/2503.01710/) (2025)
- [EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark](https://mlsanity.com/paper/2406.07162/) (2024)
- [MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation](https://mlsanity.com/paper/2412.04448/) (2024)
- [DiaASQ : A Benchmark of Conversational Aspect-based Sentiment Quadruple Analysis](https://mlsanity.com/paper/2211.05705/) (2022)
- [Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions](https://mlsanity.com/paper/2403.01222/) (2024)
- [CPED: A Large-Scale Chinese Personalized and Emotional Dialogue Dataset for Conversational AI](https://mlsanity.com/paper/2205.14727/) (2022)
- [Survey of Large Multimodal Model Datasets, Application Categories and Taxonomy](https://mlsanity.com/paper/2412.17759/) (2024)
- [Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Models](https://mlsanity.com/paper/2308.11578/) (2023)
- [Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Survey](https://mlsanity.com/paper/2406.08068/) (2024)
- [Generative Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2407.21491/) (2024)
- [Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset](https://mlsanity.com/paper/2407.02751/) (2024)
- [PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2408.09481/) (2024)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia](https://mlsanity.com/paper/2501.13306/) (2025)
- [CARAT: Contrastive Feature Reconstruction and Aggregation for Multi-Modal Multi-Label Emotion Recognition](https://mlsanity.com/paper/2312.10201/) (2023)
- [SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations](https://mlsanity.com/paper/2405.13049/) (2024)
- [Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects](https://mlsanity.com/paper/2505.20511/) (2025)
- [EmpathicStories++: A Multimodal Dataset for Empathy towards Personal Experiences](https://mlsanity.com/paper/2405.15708/) (2024)
- [BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2408.05926/) (2024)
- [TikTalk: A Video-Based Dialogue Dataset for Multi-Modal Chitchat in Real World](https://mlsanity.com/paper/2301.05880/) (2023)
- [Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling](https://mlsanity.com/paper/2410.09524/) (2024)
- [Akan Cinematic Emotions (ACE): A Multimodal Multi-party Dataset for Emotion Recognition in Movie Dialogues](https://mlsanity.com/paper/2502.10973/) (2025)
- [Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought](https://mlsanity.com/paper/2502.18186/) (2025)
- [StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs](https://mlsanity.com/paper/2509.22220/) (2025)
- [TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch](https://mlsanity.com/paper/2412.15622/) (2024)
- [M3HG: Multimodal, Multi-scale, and Multi-type Node Heterogeneous Graph for Emotion Cause Triplet Extraction in Conversations](https://mlsanity.com/paper/2508.18740/) (2025)
- [Latent Distribution Decoupling: A Probabilistic Framework for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2502.13954/) (2025)
- [VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs](https://mlsanity.com/paper/2603.08936/) (2026)
- [Object Recognition from Scientific Document based on Compartment Refinement Framework](https://mlsanity.com/paper/2312.09038/) (2023)

## Abstract (arXiv)

The emotional state of a speaker can be influenced by many different factors in dialogues, such as dialogue scene, dialogue topic, and interlocutor stimulus. The currently available data resources to support such multimodal affective analysis in dialogues are however limited in scale and diversity. In this work, we propose a Multi-modal Multi-scene Multi-label Emotional Dialogue dataset, M3ED, which contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9,082 turns and 24,449 utterances. M3 ED is annotated with 7 emotion categories (happy, surprise, sad, disgust, anger, fear, and neutral) at utterance level, and encompasses acoustic, visual, and textual modalities. To the best of our knowledge, M3ED is the first multimodal emotional dialogue dataset in Chinese. It is valuable for cross-culture emotion analysis and recognition. We apply several state-of-the-art methods on the M3ED dataset to verify the validity and quality of the dataset. We also propose a general Multimodal Dialogue-aware Interaction framework, MDI, to model the dialogue context for emotion recognition, which achieves comparable performance to the state-of-the-art methods on the M3ED. The full dataset and codes are available.