# SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video

arXiv: https://arxiv.org/abs/2210.11341 · HTML: https://mlsanity.com/paper/2210.11341/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild](https://mlsanity.com/paper/1901.02839/) (2019)
- [LiRA: Learning Visual Speech Representations from Audio through Self-supervision](https://mlsanity.com/paper/2106.09171/) (2021)
- [Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation](https://mlsanity.com/paper/1906.06196/) (2019)
- [Pre-training strategies and datasets for facial representation learning](https://mlsanity.com/paper/2103.16554/) (2021)
- [Self-supervised Contrastive Learning of Multi-view Facial Expressions](https://mlsanity.com/paper/2108.06723/) (2021)
- [Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition](https://mlsanity.com/paper/2103.13372/) (2021)
- [Spatiotemporal Contrastive Learning of Facial Expressions in Videos](https://mlsanity.com/paper/2108.03064/) (2021)

## Built on by (most cited first, 2 of 2)

- [A vector quantized masked autoencoder for audiovisual speech emotion recognition](https://mlsanity.com/paper/2305.03568/) (2023)
- [Hybrid-supervised Hypergraph-enhanced Transformer for Micro-gesture Based Emotion Recognition](https://mlsanity.com/paper/2507.14867/) (2025)

## Abstract (arXiv)

This work focuses on the apparent emotional reaction recognition (AERR) from the video-only input, conducted in a self-supervised fashion. The network is first pre-trained on different self-supervised pretext tasks and later fine-tuned on the downstream target task. Self-supervised learning facilitates the use of pre-trained architectures and larger datasets that might be deemed unfit for the target task and yet might be useful to learn informative representations and hence provide useful initializations for further fine-tuning on smaller more suitable data. Our presented contribution is two-fold: (1) an analysis of different state-of-the-art (SOTA) pretext tasks for the video-only apparent emotional reaction recognition architecture, and (2) an analysis of various combinations of the regression and classification losses that are likely to improve the performance further. Together these two contributions result in the current state-of-the-art performance for the video-only spontaneous apparent emotional reaction recognition with continuous annotations.