# UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause

arXiv: https://arxiv.org/abs/2404.00403 · HTML: https://mlsanity.com/paper/2404.00403/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 76 references that are in the map.
- First posted on arXiv 30 Mar 2024.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [DialogueRNN: An Attentive RNN for Emotion Detection in Conversations](https://mlsanity.com/paper/1811.00405/) (2018)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/1908.11540/) (2019)
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](https://mlsanity.com/paper/2010.02795/) (2020)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation](https://mlsanity.com/paper/2107.06779/) (2021)
- [Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts](https://mlsanity.com/paper/1906.01267/) (2019)
- [On Transferability of Prompt Tuning for Natural Language Processing](https://mlsanity.com/paper/2111.06719/) (2021)
- [Discovering New Intents with Deep Aligned Clustering](https://mlsanity.com/paper/2012.08987/) (2020)
- [Deep Open Intent Classification with Adaptive Decision Boundary](https://mlsanity.com/paper/2012.10209/) (2020)
- [Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection](https://mlsanity.com/paper/2106.01071/) (2021)
- [MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations](https://mlsanity.com/paper/2203.02385/) (2022)
- [EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition](https://mlsanity.com/paper/2203.13504/) (2022)
- [Multi-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/1905.05812/) (2019)
- [CoMPM: Context Modeling with Speaker's Pre-trained Memory Tracking for Emotion Recognition in Conversation](https://mlsanity.com/paper/2108.11626/) (2021)
- [Multimodal Emotion-Cause Pair Extraction in Conversations](https://mlsanity.com/paper/2110.08020/) (2021)

## Built on by (most cited first, 8 of 8)

- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Towards Multi-dimensional Explanation Alignment for Medical Classification](https://mlsanity.com/paper/2410.21494/) (2024)
- [M3HG: Multimodal, Multi-scale, and Multi-type Node Heterogeneous Graph for Emotion Cause Triplet Extraction in Conversations](https://mlsanity.com/paper/2508.18740/) (2025)
- [What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models](https://mlsanity.com/paper/2601.10159/) (2026)
- [Retrieving Implicit and Explicit Emotional Events Using Large Language Models](https://mlsanity.com/paper/2410.19128/) (2024)
- [Hierarchical Adaptive Expert for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2503.22715/) (2025)
- [Partitioner Guided Modal Learning Framework](https://mlsanity.com/paper/2507.11661/) (2025)
- [Learning Robust Pair Confidence for Multimodal Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2606.18893/) (2026)

## Abstract (arXiv)

Multimodal emotion recognition in conversation (MERC) and multimodal emotion-cause pair extraction (MECPE) have recently garnered significant attention. Emotions are the expression of affect or feelings; responses to specific events, or situations -- known as emotion causes. Both collectively explain the causality between human emotion and intents. However, existing works treat emotion recognition and emotion cause extraction as two individual problems, ignoring their natural causality. In this paper, we propose a Unified Multimodal Emotion recognition and Emotion-Cause analysis framework (UniMEEC) to explore the causality between emotion and emotion cause. Concretely, UniMEEC reformulates the MERC and MECPE tasks as mask prediction problems and unifies them with a causal prompt template. To differentiate the modal effects, UniMEEC proposes a multimodal causal prompt to probe the pre-trained knowledge specified to modality and implements cross-task and cross-modality interactions under task-oriented settings. Experiment results on four public benchmark datasets verify the model performance on MERC and MECPE tasks and achieve consistent improvements compared with the previous state-of-the-art methods.