# NUS-Emo at SemEval-2024 Task 3: Instruction-Tuning LLM for Multimodal Emotion-Cause Analysis in Conversations

arXiv: https://arxiv.org/abs/2501.17261 · HTML: https://mlsanity.com/paper/2501.17261/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 35 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://mlsanity.com/paper/2212.12017/) (2022)
- [Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Texts](https://mlsanity.com/paper/1906.01267/) (2019)
- [VPGTrans: Transfer Visual Prompt Generator across LLMs](https://mlsanity.com/paper/2305.01278/) (2023)
- [Reasoning Implicit Sentiment with Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.11255/) (2023)
- [Recognizing Emotion Cause in Conversations](https://mlsanity.com/paper/2012.11820/) (2020)
- [Multimodal Emotion-Cause Pair Extraction in Conversations](https://mlsanity.com/paper/2110.08020/) (2021)
- [DiaASQ : A Benchmark of Conversational Aspect-based Sentiment Quadruple Analysis](https://mlsanity.com/paper/2211.05705/) (2022)
- [Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination](https://mlsanity.com/paper/2305.12256/) (2023)
- [Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment](https://mlsanity.com/paper/2305.12260/) (2023)
- [SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations](https://mlsanity.com/paper/2405.13049/) (2024)
- [Joint Alignment of Multi-Task Feature and Label Spaces for Emotion Cause Pair Extraction](https://mlsanity.com/paper/2209.04112/) (2022)

## Built on by (most cited first, 5 of 5)

- [VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models](https://mlsanity.com/paper/2504.13122/) (2025)
- [Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks](https://mlsanity.com/paper/2510.19195/) (2025)
- [Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoning](https://mlsanity.com/paper/2602.00971/) (2026)
- [ReLoop: "Seeing Twice and Thinking Backwards" via Closed-loop Training to Mitigate Hallucinations in Multimodal understanding](https://mlsanity.com/paper/2507.04943/) (2025)
- [Learning Robust Pair Confidence for Multimodal Emotion-Cause Pair Extraction](https://mlsanity.com/paper/2606.18893/) (2026)

## Abstract (arXiv)

This paper describes the architecture of our system developed for Task 3 of SemEval-2024: Multimodal Emotion-Cause Analysis in Conversations. Our project targets the challenges of subtask 2, dedicated to Multimodal Emotion-Cause Pair Extraction with Emotion Category (MECPE-Cat), and constructs a dual-component system tailored to the unique challenges of this task. We divide the task into two subtasks: emotion recognition in conversation (ERC) and emotion-cause pair extraction (ECPE). To address these subtasks, we capitalize on the abilities of Large Language Models (LLMs), which have consistently demonstrated state-of-the-art performance across various natural language processing tasks and domains. Most importantly, we design an approach of emotion-cause-aware instruction-tuning for LLMs, to enhance the perception of the emotions with their corresponding causal rationales. Our method enables us to adeptly navigate the complexities of MECPE-Cat, achieving a weighted average 34.71% F1 score of the task, and securing the 2nd rank on the leaderboard. The code and metadata to reproduce our experiments are all made publicly available.