# M3T: Multi-Modal Continuous Valence-Arousal Estimation in the Wild

arXiv: https://arxiv.org/abs/2002.02957 · HTML: https://mlsanity.com/paper/2002.02957/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2020.

## Built on by (most cited first, 14 of 14)

- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion](https://mlsanity.com/paper/2107.01175/) (2021)
- [Joint Multimodal Transformer for Emotion Recognition in the Wild](https://mlsanity.com/paper/2403.10488/) (2024)
- [An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild](https://mlsanity.com/paper/2107.03465/) (2021)
- [Multitask Multi-database Emotion Recognition](https://mlsanity.com/paper/2107.04127/) (2021)
- [Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition](https://mlsanity.com/paper/2203.13285/) (2022)
- [Estimating Structural Disparities for Face Models](https://mlsanity.com/paper/2204.06562/) (2022)

## Abstract (arXiv)

This report describes a multi-modal multi-task (M3T) approach underlying our submission to the valence-arousal estimation track of the Affective Behavior Analysis in-the-wild (ABAW) Challenge, held in conjunction with the IEEE International Conference on Automatic Face and Gesture Recognition (FG) 2020. In the proposed M3T framework, we fuse both visual features from videos and acoustic features from the audio tracks to estimate the valence and arousal. The spatio-temporal visual features are extracted with a 3D convolutional network and a bidirectional recurrent neural network. Considering the correlations between valence / arousal, emotions, and facial actions, we also explores mechanisms to benefit from other tasks. We evaluated the M3T framework on the validation set provided by ABAW and it significantly outperforms the baseline method.