MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition
Not on the map. Why?
Abstract (arXiv)
This paper presents a novel approach to processing multimodal data for dynamic emotion recognition, named as the Multimodal Masked Autoencoder for Dynamic Emotion Recognition (MultiMAE-DER). The MultiMAE-DER leverages the closely correlated representation information within spatiotemporal sequences across visual and audio modalities. By utilizing a pre-trained masked autoencoder model, the MultiMAEDER is accomplished through simple, straightforward finetuning. The performance of the MultiMAE-DER is enhanced by optimizing six fusion strategies for multimodal input sequences. These strategies address dynamic feature correlations within cross-domain data across spatial, temporal, and spatiotemporal sequences. In comparison to state-of-the-art multimodal supervised learning models for dynamic emotion recognition, MultiMAE-DER enhances the weighted average recall (WAR) by 4.41% on the RAVDESS dataset and by 2.06% on the CREMAD. Furthermore, when compared with the state-of-the-art model of multimodal self-supervised learning, MultiMAE-DER achieves a 1.86% higher WAR on the IEMOCAP dataset.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 28 Apr 2024.
Builds on
26 of 38 references.
References · 26
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
- 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
- 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
- 2022Masked Autoencoders that Listencited by 271
- 2016Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Groundingcited by 243
- 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 128
14 more
- 2022MAViL: Masked Audio-Video Learnerscited by 56
- 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
- 2023MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognitioncited by 42
- 2019Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Networkcited by 37
- 2021ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognitioncited by 29
- 2022Spatio-Temporal Transformer for Dynamic Facial Expression Recognition in the Wildcited by 29
- 2017Real-time Convolutional Neural Networks for Emotion and Gender Classificationcited by 24
- 2020MSAF: Multimodal Split Attention Fusioncited by 23
- 2022Self-attention fusion for audiovisual emotion recognition with incomplete datacited by 22
- 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
- 2022A Pre-trained Audio-Visual Transformer for Emotion Recognitioncited by 12
- 2021A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognitioncited by 11
- 2023A vector quantized masked autoencoder for speech emotion recognitioncited by 8
- 2023PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognitioncited by 7
Built on by
Citing papers per year: 2024: 1; 2025: 1; 2026: 1.
3 papers.