mlsanityData to 7 Oct 2026

Emotion and Intent Joint Understanding in Multimodal Conversation: A Benchmarking Dataset

arXiv 2407.02751 ↗First posted 3 Jul 2024cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Emotion and Intent Joint Understanding in Multimodal Conversation (MC-EIU) aims to decode the semantic information manifested in a multimodal conversational history, while inferring the emotions and intents simultaneously for the current utterance. MC-EIU is enabling technology for many human-computer interfaces. However, there is a lack of available datasets in terms of annotation, modality, language diversity, and accessibility. In this work, we propose an MC-EIU dataset, which features 7 emotion categories, 9 intent categories, 3 modalities, i.e., textual, acoustic, and visual content, and two languages, i.e., English and Mandarin. Furthermore, it is completely open-source for free access. To our knowledge, MC-EIU is the first comprehensive and rich emotion and intent joint understanding dataset for multimodal conversation. Together with the release of the dataset, we also develop an Emotion and Intent Interaction (EI2) network as a reference system by modeling the deep correlation between emotion and intent in the multimodal conversation. With comparative experiments and ablation studies, we demonstrate the effectiveness of the proposed EI2 method on the MC-EIU dataset. The dataset and codes will be made available at: https://github.com/MC-EIU/MC-EIU.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 21 of its 47 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

21 of 47 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  4. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  5. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  6. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  7. 2020Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Casecited by 129
  8. 2014Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognitioncited by 111
  9. 2018EmotionLines: An Emotion Corpus of Multi-Party Conversationscited by 80
  10. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  11. 2022DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speechcited by 67
  12. 2018The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systemscited by 61
9 more
  1. 2020A Taxonomy of Empathetic Response Intents in Human Social Conversationscited by 51
  2. 2016Trace Norm Regularised Deep Multi-Task Learningcited by 50
  3. 2022M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Databasecited by 50
  4. 2022MIntRec: A New Dataset for Multimodal Intent Recognitioncited by 41
  5. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  6. 2023Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modelingcited by 17
  7. 2022Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalitiescited by 16
  8. 2021Rethinking Hard-Parameter Sharing in Multi-Domain Learningcited by 6
  9. 2020Fine-grained Emotion and Intent Learning in Movie Dialoguescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  2. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  3. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Surveycited by 19
  4. 2025MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Modelscited by 17
  5. 2025Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmarkcited by 8
  6. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  7. 2026OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computingcited by 6
  8. 2025Deep Learning Approaches for Multimodal Intent Recognition: A Surveycited by 5
  9. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understandingcited by 5
  10. 2025VIBE: Can a VLM Read the Room?cited by 4
  11. 2025Fitting Different Interactive Information: Joint Classification of Emotion and Intentioncited by 0
  12. 2025DialogGraph-LLM: Graph-Informed LLMs for End-to-End Audio Dialogue Intent Recognitioncited by 0
3 more
  1. 2026Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoningcited by 0
  2. 2026MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agentscited by 0
  3. 2026MISID: A Multimodal Multi-turn Dataset for Complex Intent Recognition in Strategic Deception Gamescited by 0