# Two-Stream Aural-Visual Affect Analysis in the Wild

arXiv: https://arxiv.org/abs/2002.03399 · HTML: https://mlsanity.com/paper/2002.03399/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2020.

## Built on by (most cited first, 30 of 51)

- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [NR-DFERNet: Noise-Robust Network for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2206.04975/) (2022)
- [A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition](https://mlsanity.com/paper/2107.04187/) (2021)
- [Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition](https://mlsanity.com/paper/2303.08356/) (2023)
- [A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis](https://mlsanity.com/paper/2107.04225/) (2021)
- [Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition](https://mlsanity.com/paper/2107.03708/) (2021)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [Coarse-to-Fine Cascaded Networks with Smooth Predicting for Video Facial Expression Recognition](https://mlsanity.com/paper/2203.13052/) (2022)
- [Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition](https://mlsanity.com/paper/2103.13372/) (2021)
- [Recursive Joint Cross-Modal Attention for Multimodal Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2403.13659/) (2024)
- [Joint Multimodal Transformer for Emotion Recognition in the Wild](https://mlsanity.com/paper/2403.10488/) (2024)
- [An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild](https://mlsanity.com/paper/2107.03465/) (2021)
- [Multitask Multi-database Emotion Recognition](https://mlsanity.com/paper/2107.04127/) (2021)
- [Multi-modal Multi-label Facial Action Unit Detection with Transformer](https://mlsanity.com/paper/2203.13301/) (2022)
- [An Ensemble Approach for Facial Expression Analysis in Video](https://mlsanity.com/paper/2203.12891/) (2022)
- [Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild](https://mlsanity.com/paper/2407.12927/) (2024)
- [Causal affect prediction model using a facial image sequence](https://mlsanity.com/paper/2107.03886/) (2021)
- [MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network](https://mlsanity.com/paper/2503.12623/) (2025)
- [Continuous-Time Audiovisual Fusion with Recurrence vs. Attention for In-The-Wild Affect Recognition](https://mlsanity.com/paper/2203.13285/) (2022)
- [Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition](https://mlsanity.com/paper/2304.07958/) (2023)
- [Boosting Continuous Emotion Recognition with Self-Pretraining using Masked Autoencoders, Temporal Convolutional Networks, and Transformers](https://mlsanity.com/paper/2403.11440/) (2024)
- [Emotion Recognition with CLIP and Sequential Learning](https://mlsanity.com/paper/2503.09929/) (2025)

## Abstract (arXiv)

Human affect recognition is an essential part of natural human-computer interaction. However, current methods are still in their infancy, especially for in-the-wild data. In this work, we introduce our submission to the Affective Behavior Analysis in-the-wild (ABAW) 2020 competition. We propose a two-stream aural-visual analysis model to recognize affective behavior from videos. Audio and image streams are first processed separately and fed into a convolutional neural network. Instead of applying recurrent architectures for temporal analysis we only use temporal convolutions. Furthermore, the model is given access to additional features extracted during face-alignment. At training time, we exploit correlations between different emotion representations to improve performance. Our model achieves promising results on the challenging Aff-Wild2 database.