mlsanityData to 14 Oct 2026

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

arXiv 2408.10500 ↗First posted 20 Aug 2024cs.MM cs.CV cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper presents our winning approach for the MER-NOISE and MER-OV tracks of the MER2024 Challenge on multimodal emotion recognition. Our system leverages the advanced emotional understanding capabilities of Emotion-LLaMA to generate high-quality annotations for unlabeled samples, addressing the challenge of limited labeled data. To enhance multimodal fusion while mitigating modality-specific noise, we introduce Conv-Attention, a lightweight and efficient hybrid framework. Extensive experimentation vali-dates the effectiveness of our approach. In the MER-NOISE track, our system achieves a state-of-the-art weighted average F-score of 85.30%, surpassing the second and third-place teams by 1.47% and 1.65%, respectively. For the MER-OV track, our utilization of Emotion-LLaMA for open-vocabulary annotation yields an 8.52% improvement in average accuracy and recall compared to GPT-4V, securing the highest score among all participating large multimodal models. The code and model for Emotion-LLaMA are available at https://github.com/ZebangCheng/Emotion-LLaMA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 63 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

36 of 63 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
24 more
  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  5. 2021CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classificationcited by 806
  6. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  7. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  8. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  10. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  11. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  12. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 266
  13. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  14. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  15. 2020DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wildcited by 128
  16. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 121
  17. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  18. 2021Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolutioncited by 52
  19. 2024EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuningcited by 43
  20. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  21. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
  22. 2023Learning Emotion Representations from Verbal and Nonverbal Communicationcited by 31
  23. 2020Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognitioncited by 24
  24. 2024MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Modelscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding
  2. 2025Pioneering Multimodal Emotion Recognition in the Era of Large Models: From Closed Sets to Open Vocabularies
  3. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
  4. 2026Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?
  5. 2026MER 2026: From Discriminative Emotion Recognition to Generative Emotion Understanding
  6. 2026MVFA: A Multi-View Text-Guided Multimodal Fusion LLM Adapter for Sentiment Analysis and Emotion Recognition
  7. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
  8. 2026EmoMM: Benchmarking and Steering MLLM for Multimodal Emotion Recognition under Conflict and Missingness