Interactive Multimodal Fusion with Temporal Modeling
Not on the map. Why?
Abstract (arXiv)
This paper presents our method for the estimation of valence-arousal (VA) in the 8th Affective Behavior Analysis in-the-Wild (ABAW) competition. Our approach integrates visual and audio information through a multimodal framework. The visual branch uses a pre-trained ResNet model to extract spatial features from facial images. The audio branches employ pre-trained VGG models to extract VGGish and LogMel features from speech signals. These features undergo temporal modeling using Temporal Convolutional Networks (TCNs). We then apply cross-modal attention mechanisms, where visual features interact with audio features through query-key-value attention structures. Finally, the features are concatenated and passed through a regression layer to predict valence and arousal. Our method achieves competitive performance on the Aff-Wild2 dataset, demonstrating effective multimodal fusion for VA estimation in-the-wild.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 13 Mar 2025.
Builds on
23 of 37 references.
References · 23
- 2017Attention Is All You Needcited by 60,361
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
- 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
- 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
- 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
- 2020Analysing Affective Behavior in the First ABAW 2020 Competitioncited by 178
- 2021Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Frameworkcited by 174
- 2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
- 2021Analysing Affective Behavior in the second ABAW2 Competitioncited by 163
- 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
11 more
- 2022ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challengescited by 156
- 2019Face Behavior a la carte: Expressions, Affect and Action Units in a Single Networkcited by 155
- 2024The 6th Affective Behavior Analysis in-the-wild (ABAW) Competitioncited by 78
- 2014DenseNet: Implementing Efficient ConvNet Descriptor Pyramidscited by 66
- 2024Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyondcited by 60
- 20247th ABAW Competition: Multi-Task Learning and Compound Expression Recognitioncited by 53
- 2020Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attentioncited by 37
- 2021Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognitioncited by 19
- 2021Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusioncited by 15
- 2018Using Deep Autoencoders for Facial Expression Recognitioncited by 7
- 2024Multimodal Fusion Method with Spatiotemporal Sequences and Relationship Learning for Valence-Arousal Estimationcited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.