# An Enhanced Adversarial Network with Combined Latent Features for Spatio-Temporal Facial Affect Estimation in the Wild

arXiv: https://arxiv.org/abs/2102.09150 · HTML: https://mlsanity.com/paper/2102.09150/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 44 references that are in the map.
- First posted on arXiv 18 Feb 2021.

## Builds on (most cited first, 13 of 13)

- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.03650/) (2016)
- [SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild](https://mlsanity.com/paper/1901.02839/) (2019)
- [Towards Diverse and Natural Image Descriptions via a Conditional GAN](https://mlsanity.com/paper/1703.06029/) (2017)
- [YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers](https://mlsanity.com/paper/1811.05588/) (2018)
- [The OMG-Emotion Behavior Dataset](https://mlsanity.com/paper/1803.05434/) (2018)
- [Weighted Speech Distortion Losses for Neural-network-based Real-time Speech Enhancement](https://mlsanity.com/paper/2001.10601/) (2020)
- [Two-level Attention with Two-stage Multi-task Learning for Facial Emotion Recognition](https://mlsanity.com/paper/1811.12139/) (2018)
- [End-to-end facial and physiological model for Affective Computing and applications](https://mlsanity.com/paper/1912.04711/) (2019)

## Built on by (most cited first, 1 of 1)

- [Machine Learning-based Lie Detector applied to a Novel Annotated Game Dataset](https://mlsanity.com/paper/2104.12345/) (2021)

## Abstract (arXiv)

Affective Computing has recently attracted the attention of the research community, due to its numerous applications in diverse areas. In this context, the emergence of video-based data allows to enrich the widely used spatial features with the inclusion of temporal information. However, such spatio-temporal modelling often results in very high-dimensional feature spaces and large volumes of data, making training difficult and time consuming. This paper addresses these shortcomings by proposing a novel model that efficiently extracts both spatial and temporal features of the data by means of its enhanced temporal modelling based on latent features. Our proposed model consists of three major networks, coined Generator, Discriminator, and Combiner, which are trained in an adversarial setting combined with curriculum learning to enable our adaptive attention modules. In our experiments, we show the effectiveness of our approach by reporting our competitive results on both the AFEW-VA and SEWA datasets, suggesting that temporal modelling improves the affect estimates both in qualitative and quantitative terms. Furthermore, we find that the inclusion of attention mechanisms leads to the highest accuracy improvements, as its weights seem to correlate well with the appearance of facial movements, both in terms of temporal localisation and intensity. Finally, we observe the sequence length of around 160,ms to be the optimum one for temporal modelling, which is consistent with other relevant findings utilising similar lengths.