mlsanityData to 7 Oct 2026

Efficient Feature Extraction and Late Fusion Strategy for Audiovisual Emotional Mimicry Intensity Estimation

arXiv 2403.11757 ↗First posted 18 Mar 2024cs.MM cs.LG cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

In this paper, we present the solution to the Emotional Mimicry Intensity (EMI) Estimation challenge, which is part of 6th Affective Behavior Analysis in-the-wild (ABAW) Competition.The EMI Estimation challenge task aims to evaluate the emotional intensity of seed videos by assessing them from a set of predefined emotion categories (i.e., "Admiration", "Amusement", "Determination", "Empathic Pain", "Excitement" and "Joy"). To tackle this challenge, we extracted rich dual-channel visual features based on ResNet18 and AUs for the video modality and effective single-channel features based on Wav2Vec2.0 for the audio modality. This allowed us to obtain comprehensive emotional features for the audiovisual modality. Additionally, leveraging a late fusion strategy, we averaged the predictions of the visual and acoustic models, resulting in a more accurate estimation of audiovisual emotional mimicry intensity. Experimental results validate the effectiveness of our approach, with the average Pearson's correlation Coefficient(ρ) across the 6 emotion dimensionson the validation set achieving 0.3288.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 21 of its 61 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

21 of 61 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895
  9. 2017AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wildcited by 490
  10. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 301
  11. 2020Analysing Affective Behavior in the First ABAW 2020 Competitioncited by 178
  12. 2021Analysing Affective Behavior in the second ABAW2 Competitioncited by 163
9 more
  1. 2022ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challengescited by 156
  2. 2022ABAW: Learning from Synthetic Data & Multi-Task Learning Challengescited by 110
  3. 2023ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challengescited by 110
  4. 2024The 6th Affective Behavior Analysis in-the-wild (ABAW) Competitioncited by 78
  5. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76
  6. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 52
  7. 2021Multimodal Learning using Optimal Transport for Sarcasm and Humor Detectioncited by 36
  8. 2020ColBERT: Using BERT Sentence Embedding in Parallel Neural Networks for Computational Humorcited by 18
  9. 2022Hybrid Multimodal Fusion for Humor Detectioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2025HSEmotion Team at ABAW-8 Competition: Audiovisual Ambivalence/Hesitancy, Emotional Mimicry Intensity and Facial Expression Recognitioncited by 15
  2. 2024UniMat: Unifying Materials Embeddings through Multi-modal Learningcited by 6
  3. 2025Semantic Matters: Multimodal Features for Affective Analysiscited by 5
  4. 2026Solution for 10th Competition on Ambivalence/Hesitancy (AH) Video Recognition Challenge using Divergence-Based Multimodal Fusioncited by 4
  5. 2026Multimodal Emotion Regression with Multi-Objective Optimization and VAD-Aware Audio Modeling for the 10th ABAW EMI Trackcited by 2
  6. 2026Two-Stage Multimodal Framework for Emotion Mimicry Intensity Predictioncited by 0