# Multimodal Emotion-Cause Pair Extraction in Conversations

arXiv: https://arxiv.org/abs/2110.08020 · HTML: https://mlsanity.com/paper/2110.08020/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 71 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs](https://mlsanity.com/paper/2010.05953/) (2020)
- [TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://mlsanity.com/paper/1904.11574/) (2019)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts](https://mlsanity.com/paper/1906.01267/) (2019)
- [EmotionLines: An Emotion Corpus of Multi-Party Conversations](https://mlsanity.com/paper/1802.08379/) (2018)
- [Recognizing Emotion Cause in Conversations](https://mlsanity.com/paper/2012.11820/) (2020)
- [GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception](https://mlsanity.com/paper/1912.03184/) (2019)
- [RTHN: A RNN-Transformer Hierarchical Network for Emotion Cause Extraction](https://mlsanity.com/paper/1906.01236/) (2019)
- [A Question Answering Approach to Emotion Cause Extraction](https://mlsanity.com/paper/1708.05482/) (2017)
- [From Independent Prediction to Re-ordered Prediction: Integrating Relative Position and Global Label Information to Emotion Cause Identification](https://mlsanity.com/paper/1906.01230/) (2019)
- [Detecting Concept-level Emotion Cause in Microblogging](https://mlsanity.com/paper/1504.08050/) (2015)

## Built on by (most cited first, 30 of 31)

- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations](https://mlsanity.com/paper/2405.13049/) (2024)
- [MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Models](https://mlsanity.com/paper/2404.00511/) (2024)
- [UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause](https://mlsanity.com/paper/2404.00403/) (2024)
- [SemEval 2024 -- Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF)](https://mlsanity.com/paper/2402.18944/) (2024)
- [NUS-Emo at SemEval-2024 Task 3: Instruction-Tuning LLM for Multimodal Emotion-Cause Analysis in Conversations](https://mlsanity.com/paper/2501.17261/) (2024)
- [Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2606.08566/) (2026)
- [M3HG: Multimodal, Multi-scale, and Multi-type Node Heterogeneous Graph for Emotion Cause Triplet Extraction in Conversations](https://mlsanity.com/paper/2508.18740/) (2025)
- [ECQED: Emotion-Cause Quadruple Extraction in Dialogs](https://mlsanity.com/paper/2306.03969/) (2023)
- [JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models](https://mlsanity.com/paper/2403.04798/) (2024)
- [nicolay-r at SemEval-2024 Task 3: Using Flan-T5 for Reasoning Emotion Cause in Conversations with Chain-of-Thought on Emotion States](https://mlsanity.com/paper/2404.03361/) (2024)
- [PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations](https://mlsanity.com/paper/2404.05502/) (2024)
- [Samsung Research China-Beijing at SemEval-2024 Task 3: A multi-stage framework for Emotion-Cause Pair Extraction in Conversations](https://mlsanity.com/paper/2404.16905/) (2024)
- [Discourse-Aware Emotion Cause Extraction in Conversations](https://mlsanity.com/paper/2210.14419/) (2022)
- [LastResort at SemEval-2024 Task 3: Exploring Multimodal Emotion Cause Pair Extraction as Sequence Labelling Task](https://mlsanity.com/paper/2404.02088/) (2024)
- [LyS at SemEval-2024 Task 3: An Early Prototype for End-to-End Multimodal Emotion Linking as Graph-Based Parsing](https://mlsanity.com/paper/2405.06483/) (2024)
- [Think out Loud: Emotion Deducing Explanation in Dialogues](https://mlsanity.com/paper/2406.04758/) (2024)
- [Generation or Judgement? A Paradigm Perspective on LLM-Based Emotion-Cause Pair Extraction in Conversation](https://mlsanity.com/paper/2607.26967/) (2026)
- [MasonTigers at SemEval-2024 Task 10: Emotion Discovery and Flip Reasoning in Conversation with Ensemble of Transformers and Prompting](https://mlsanity.com/paper/2407.00581/) (2024)
- [Generative Emotion Cause Explanation in Multimodal Conversations](https://mlsanity.com/paper/2411.02430/) (2024)
- [AIMA at SemEval-2024 Task 3: Simple Yet Powerful Emotion Cause Pair Analysis](https://mlsanity.com/paper/2501.11170/) (2025)
- [Span-level Emotion-Cause-Category Triplet Extraction with Instruction Tuning LLMs and Data Augmentation](https://mlsanity.com/paper/2504.12331/) (2025)
- [Analysis of Voluntarily Reported Data Post Mesh Implantation for Detecting Public Emotion and Identifying Concern Reports](https://mlsanity.com/paper/2509.04517/) (2025)
- [MCIHN: A Hybrid Network Model Based on Multi-path Cross-modal Interaction for Multimodal Emotion Recognition](https://mlsanity.com/paper/2510.24827/) (2025)
- [MISID: A Multimodal Multi-turn Dataset for Complex Intent Recognition in Strategic Deception Games](https://mlsanity.com/paper/2604.12700/) (2026)
- [Emotion-Cause Pair Extraction in Conversations via Semantic Decoupling and Graph Alignment](https://mlsanity.com/paper/2604.19547/) (2026)
- [They Are Not the Same: Direct Causes Are Not Grounded Emotion Explanations](https://mlsanity.com/paper/2605.25208/) (2026)
- [Learning Robust Pair Confidence for Multimodal Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2606.18893/) (2026)
- [AffectReveal: Event-Grounded Emotion Recognition Beyond Visual Appearances](https://mlsanity.com/paper/2609.36563/) (2026)
- [Causal-EVC: Breaking Emotional Spurious Causality via Spatiotemporal Grounding and Counterfactual Intervention](https://mlsanity.com/paper/2609.36776/) (2026)

## Abstract (arXiv)

Emotion cause analysis has received considerable attention in recent years. Previous studies primarily focused on emotion cause extraction from texts in news articles or microblogs. It is also interesting to discover emotions and their causes in conversations. As conversation in its natural form is multimodal, a large number of studies have been carried out on multimodal emotion recognition in conversations, but there is still a lack of work on multimodal emotion cause analysis. In this work, we introduce a new task named Multimodal Emotion-Cause Pair Extraction in Conversations, aiming to jointly extract emotions and their associated causes from conversations reflected in multiple modalities (text, audio and video). We accordingly construct a multimodal conversational emotion cause dataset, Emotion-Cause-in-Friends, which contains 9,272 multimodal emotion-cause pairs annotated on 13,509 utterances in the sitcom Friends. We finally benchmark the task by establishing a baseline system that incorporates multimodal features for emotion-cause pair extraction. Preliminary experimental results demonstrate the potential of multimodal information fusion for discovering both emotions and causes in conversations.