# EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations

arXiv: https://arxiv.org/abs/2505.23018 · HTML: https://mlsanity.com/paper/2505.23018/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2025.

## Built on by (most cited first, 8 of 8)

- [Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs](https://mlsanity.com/paper/2512.16378/) (2025)
- [AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning](https://mlsanity.com/paper/2601.06086/) (2025)
- [VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs](https://mlsanity.com/paper/2603.08936/) (2026)
- [AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models](https://mlsanity.com/paper/2610.01560/) (2026)
- [SpEmoC: A Balanced Speaker-Segment Multimodal Emotion Benchmark](https://mlsanity.com/paper/2607.18109/) (2026)
- [Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens](https://mlsanity.com/paper/2607.26350/) (2026)
- [EmotionDialogCN: A Spontaneous Multimodal Dataset for Mandarin Emotional Dialogue](https://mlsanity.com/paper/2608.20905/) (2026)
- [AphaVoice: Text-to-Speech Model for Aphasia Patient Simulation](https://mlsanity.com/paper/2610.14479/) (2026)

## Abstract (arXiv)

In recent years, emotion recognition plays a critical role in applications such as human-computer interaction, mental health monitoring, and sentiment analysis. While datasets for emotion analysis in languages such as English have proliferated, there remains a pressing need for high-quality, comprehensive datasets tailored to the unique linguistic, cultural, and multimodal characteristics of Chinese. In this work, we propose EmotionTalk, an interactive Chinese multimodal emotion dataset with rich annotations. This dataset provides multimodal information from 19 actors participating in dyadic conversational settings, incorporating acoustic, visual, and textual modalities. It includes 23.6 hours of speech (19,250 utterances), annotations for 7 utterance-level emotion categories (happy, surprise, sad, disgust, anger, fear, and neutral), 5-dimensional sentiment labels (negative, weakly negative, neutral, weakly positive, and positive) and 4-dimensional speech captions (speaker, speaking style, emotion and overall). The dataset is well-suited for research on unimodal and multimodal emotion recognition, missing modality challenges, and speech captioning tasks. To our knowledge, it represents the first high-quality and versatile Chinese dialogue multimodal emotion dataset, which is a valuable contribution to research on cross-cultural emotion analysis and recognition. Additionally, we conduct experiments on EmotionTalk to demonstrate the effectiveness and quality of the dataset. It will be open-source and freely available for all academic purposes. The dataset and codes will be made available at: https://github.com/NKU-HLT/EmotionTalk.