mlsanityData to 7 Oct 2026

Leveraging Contrastive Learning and Self-Training for Multimodal Emotion Recognition with Limited Labeled Samples

arXiv 2409.04447 ↗First posted 23 Aug 2024cs.SD cs.AI eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

The Multimodal Emotion Recognition challenge MER2024 focuses on recognizing emotions using audio, language, and visual signals. In this paper, we present our submission solutions for the Semi-Supervised Learning Sub-Challenge (MER2024-SEMI), which tackles the issue of limited annotated data in emotion recognition. Firstly, to address the class imbalance, we adopt an oversampling strategy. Secondly, we propose a modality representation combinatorial contrastive learning (MR-CCL) framework on the trimodal input data to establish robust initial models. Thirdly, we explore a self-training approach to expand the training set. Finally, we enhance prediction robustness through a multi-classifier weighted soft voting strategy. Our proposed method is validated to be effective on the MER2024-SEMI Challenge, achieving a weighted average F-score of 88.25% and ranking 6th on the leaderboard. Our project is available at https://github.com/WooyoohL/MER2024-SEMI.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 19 of its 36 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

19 of 36 references.

References · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  8. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 123
  9. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  10. 2023VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understandingcited by 62
  11. 2023MmAP : Multi-modal Alignment Prompt for Cross-domain Multi-task Learningcited by 41
  12. 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
7 more
  1. 2024Credit Card Fraud Detection Using Advanced Transformer Modelcited by 26
  2. 2024Harnessing XGBoost for Robust Biomarker Selection of Obsessive-Compulsive Disorder (OCD) from Adolescent Brain Cognitive Development (ABCD) datacited by 26
  3. 2022Analyzing Modality Robustness in Multimodal Sentiment Analysiscited by 20
  4. 2024Enhanced Detection Classification via Clustering SVM for Various Robot Collaboration Taskcited by 17
  5. 2022Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalitiescited by 16
  6. 2023ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMscited by 13
  7. 2019Attending to Emotional Narrativescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Twin Co-Adaptive Dialogue for Progressive Image Generationcited by 11
  2. 2025More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignmentcited by 4
  3. 2025TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generationcited by 1
  4. 2025Fitting Different Interactive Information: Joint Classification of Emotion and Intentioncited by 0
  5. 2026InCarEmo: A Multimodal Dataset for In-Cabin Emotion Recognition and Driver State Monitoringcited by 0