Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better
Not on the map. Why?
Abstract (arXiv)
In this paper, we present our solutions for emotion recognition in the sub-challenges of Multimodal Emotion Recognition Challenge (MER2024). To mitigate the modal competition issue between audio and text, we adopt an early fusion strategy based on a large language model, where joint training of audio and text is conducted initially. And the joint Audio-Text modal feature will be late-fused with other unimodal features. In order to solve the problems of data insufficiency and class imbalance, We use multiple turns of multi-model voting for data mining. Moreover, to enhance the quality of audio features, we employ speech source separation to preprocess audios. Our model ranks 2nd in both MER2024-SEMI and MER2024-NOISE, validating our method's effectiveness.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 12 Sep 2024.
Builds on
29 of 39 references.
References · 29
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2014Deep Learning Face Attributes in the Wildcited by 3,679
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
- 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
17 more
- 2023Baichuan 2: Open Large-scale Language Modelscited by 664
- 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
- 2015WIDER FACE: A Face Detection Benchmarkcited by 297
- 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
- 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 139
- 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
- 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 92
- 2018Multi-attention Recurrent Network for Human Communication Comprehensioncited by 89
- 2022UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognitioncited by 61
- 2022A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advancescited by 59
- 2018Building and Evaluation of a Real Room Impulse Response Datasetcited by 43
- 2023MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separationcited by 39
- 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
- 2023MER 2023: Multi-label Learning, Modality Robustness, and Semi-Supervised Learningcited by 33
- 2024MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognitioncited by 27
- 2015Learning Social Relation Traits from Face Imagescited by 25
- 2024MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognitioncited by 23
Built on by
Citing papers per year: 2025: 1.
1 paper.