mlsanityData to 7 Oct 2026

A Pre-trained Audio-Visual Transformer for Emotion Recognition

arXiv 2201.09165 ↗First posted 23 Jan 2022cs.MM cs.CV cs.SDcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a pretrained audio-visual Transformer trained on more than 500k utterances from nearly 4000 celebrities from the VoxCeleb2 dataset for human behavior understanding. The model aims to capture and extract useful information from the interactions between human facial and auditory behaviors, with application in emotion recognition. We evaluate the model performance on two datasets, namely CREMAD-D (emotion classification) and MSP-IMPROV (continuous emotion regression). Experimental results show that fine-tuning the pre-trained model helps improving emotion classification accuracy by 5-7% and Concordance Correlation Coefficients (CCC) in continuous emotion recognition by 0.03-0.09 compared to the same model trained from scratch. We also demonstrate the robustness of finetuning the pre-trained model in a low-resource setting. With only 10% of the original training set provided, fine-tuning the pre-trained model can lead to at least 10% better emotion recognition accuracy and a CCC score improvement by at least 0.1 for continuous emotion recognition.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 23 Jan 2022.

Builds on

21 of 29 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  9. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  10. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  11. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  12. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
9 more
  1. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  2. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  3. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  4. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  5. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 58
  6. 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
  7. 2020Multitask Learning and Multistage Fusion for Dimensional Audiovisual Emotion Recognitioncited by 9
  8. 2020Deep Multilayer Perceptrons for Dimensional Speech Emotion Recognitioncited by 1
  9. 2020Temporal aggregation of audio-visual modalities for emotion recognitioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 2; 2026: 1.

12 papers.

  1. 2023MAE-DFER: Efficient Masked Autoencoder for Self-supervised Dynamic Facial Expression Recognitioncited by 42
  2. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  3. 2023SVFAP: Self-supervised Video Facial Affect Perceivercited by 18
  4. 2024Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognitioncited by 15
  5. 2024Joint Multimodal Transformer for Emotion Recognition in the Wildcited by 13
  6. 2024Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wildcited by 9
  7. 2023A vector quantized masked autoencoder for audiovisual speech emotion recognitioncited by 6
  8. 2023Versatile audio-visual learning for emotion recognitioncited by 5
  9. 2024MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognitioncited by 3
  10. 2025eMotions: A Large-Scale Dataset and Audio-Visual Fusion Network for Emotion Analysis in Short-form Videoscited by 1
  11. 2026Multimodal Emotion Recognition via Bi-directional Cross-Attention and Temporal Modelingcited by 1
  12. 2025United we stand, Divided we fall: Handling Weak Complementary Relationships for Audio-Visual Emotion Recognition in Valence-Arousal Spacecited by 0