# Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond

arXiv: https://arxiv.org/abs/1804.10938 · HTML: https://mlsanity.com/paper/1804.10938/

## Key facts

- Cited by 216 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2018.

## Built on by (most cited first, 30 of 216)

- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [FERV39k: A Large-Scale Multi-Scene Dataset for Facial Expression Recognition in Videos](https://mlsanity.com/paper/2203.09463/) (2022)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [Attribute-preserving Face Dataset Anonymization via Latent Code Optimization](https://mlsanity.com/paper/2303.11296/) (2023)
- [A Deep Neural Architecture for Harmonizing 3-D Input Data Analysis and Decision Making in Medical Imaging](https://mlsanity.com/paper/2303.00175/) (2023)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [MixAugment & Mixup: Augmentation Methods for Facial Expression Recognition](https://mlsanity.com/paper/2205.04442/) (2022)
- [Spatio-Temporal Transformer for Dynamic Facial Expression Recognition in the Wild](https://mlsanity.com/paper/2205.04749/) (2022)
- [Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos](https://mlsanity.com/paper/2112.00585/) (2021)
- [A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition](https://mlsanity.com/paper/2107.04187/) (2021)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [COVID-19 Computer-aided Diagnosis through AI-assisted CT Imaging Analysis: Deploying a Medical AI System](https://mlsanity.com/paper/2403.06242/) (2024)
- [The Multimodal Sentiment Analysis in Car Reviews (MuSe-CaR) Dataset: Collection, Insights and Improvements](https://mlsanity.com/paper/2101.06053/) (2021)
- [Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition](https://mlsanity.com/paper/2108.04228/) (2021)
- [Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3](https://mlsanity.com/paper/2203.13031/) (2022)
- [AI-MIA: COVID-19 Detection & Severity Analysis through Medical Imaging](https://mlsanity.com/paper/2206.04732/) (2022)
- [Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition](https://mlsanity.com/paper/2303.08356/) (2023)
- [BTDNet: a Multi-Modal Approach for Brain Tumor Radiogenomic Classification](https://mlsanity.com/paper/2310.03485/) (2023)
- [A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis](https://mlsanity.com/paper/2107.04225/) (2021)
- [Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition](https://mlsanity.com/paper/2107.03708/) (2021)
- [Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention](https://mlsanity.com/paper/2209.09068/) (2022)
- [Affective Behaviour Analysis via Integrating Multi-Modal Knowledge](https://mlsanity.com/paper/2403.10825/) (2024)

## Abstract (arXiv)

Automatic understanding of human affect using visual signals is of great importance in everyday human-machine interactions. Appraising human emotional states, behaviors and reactions displayed in real-world settings, can be accomplished using latent continuous dimensions (e.g., the circumplex model of affect). Valence (i.e., how positive or negative is an emotion) & arousal (i.e., power of the activation of the emotion) constitute popular and effective affect representations. Nevertheless, the majority of collected datasets this far, although containing naturalistic emotional states, have been captured in highly controlled recording conditions. In this paper, we introduce the Aff-Wild benchmark for training and evaluating affect recognition algorithms. We also report on the results of the First Affect-in-the-wild Challenge that was organized in conjunction with CVPR 2017 on the Aff-Wild database and was the first ever challenge on the estimation of valence and arousal in-the-wild. Furthermore, we design and extensively train an end-to-end deep neural architecture which performs prediction of continuous emotion dimensions based on visual cues. The proposed deep learning architecture, AffWildNet, includes convolutional & recurrent neural network layers, exploiting the invariant properties of convolutional features, while also modeling temporal dynamics that arise in human behavior via the recurrent layers. The AffWildNet produced state-of-the-art results on the Aff-Wild Challenge. We then exploit the AffWild database for learning features, which can be used as priors for achieving best performances both for dimensional, as well as categorical emotion recognition, using the RECOLA, AFEW-VA and EmotiW datasets, compared to all other methods designed for the same goal. The database and emotion recognition models are available at http://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge.