mlsanityData to 7 Oct 2026

MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Models

arXiv 2404.00511 ↗First posted 31 Mar 2024cs.CL cs.CV cs.MMcited by 9

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

This paper presents our winning submission to Subtask 2 of SemEval 2024 Task 3 on multimodal emotion cause analysis in conversations. We propose a novel Multimodal Emotion Recognition and Multimodal Emotion Cause Extraction (MER-MCE) framework that integrates text, audio, and visual modalities using specialized emotion encoders. Our approach sets itself apart from top-performing teams by leveraging modality-specific features for enhanced emotion understanding and causality inference. Experimental evaluation demonstrates the advantages of our multimodal approach, with our submission achieving a competitive weighted F1 score of 0.3435, ranking third with a margin of only 0.0339 behind the 1st team and 0.0025 behind the 2nd team. Project: https://github.com/MIPS-COLT/MER-MCE.git

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 24 references that are in the map. First posted on arXiv 31 Mar 2024.

Builds on

16 of 24 references.

References · 16

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  10. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  12. 2019Emotion-Cause Pair Extraction: A New Task to Emotion Analysis in Textscited by 82
4 more
  1. 2022EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognitioncited by 48
  2. 2021Multimodal Emotion-Cause Pair Extraction in Conversationscited by 30
  3. 2024FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audiocited by 20
  4. 2023EmotionIC: emotional inertia and contagion-driven dependency modeling for emotion recognition in conversationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 3; 2026: 1.

9 papers.

  1. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 117
  2. 2024Multimodal Multi-turn Conversation Stance Detection: A Challenge Dataset and Effective Modelcited by 12
  3. 2024SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversationscited by 11
  4. 2025Why We Feel: Breaking Boundaries in Emotional Reasoning with Multimodal Large Language Modelscited by 4
  5. 2024POPoS: Improving Efficient and Robust Facial Landmark Detection with Parallel Optimal Position Searchcited by 2
  6. 2025When One Modality Sabotages the Others: A Diagnostic Lens on Multimodal Reasoningcited by 2
  7. 2024SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognitioncited by 0
  8. 2025PRISM of Opinions: A Persona-Reasoned Multimodal Framework for User-centric Conversational Stance Detectioncited by 0
  9. 2026Learning Robust Pair Confidence for Multimodal Emotion-Cause Pair Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
  2. 2025Pioneering Multimodal Emotion Recognition in the Era of Large Models: From Closed Sets to Open Vocabularies
  3. 2024EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding
  4. 2026MER 2026: From Discriminative Emotion Recognition to Generative Emotion Understanding
  5. 2025Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
  6. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
  7. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding
  8. 2026Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?