# Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition

arXiv: https://arxiv.org/abs/2103.13372 · HTML: https://mlsanity.com/paper/2103.13372/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 76 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [RetinaFace: Single-stage Dense Face Localisation in the Wild](https://mlsanity.com/paper/1905.00641/) (2019)
- [Convolutional Conditional Neural Processes](https://mlsanity.com/paper/1910.13556/) (2019)
- [Exact Gaussian Processes on a Million Data Points](https://mlsanity.com/paper/1903.08114/) (2019)
- [AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition](https://mlsanity.com/paper/1907.11510/) (2019)
- [Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition](https://mlsanity.com/paper/1811.07770/) (2018)
- [Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes](https://mlsanity.com/paper/2007.01332/) (2020)
- [SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild](https://mlsanity.com/paper/1901.02839/) (2019)
- [Sequential Neural Processes](https://mlsanity.com/paper/1906.10264/) (2019)
- [Two-Stream Aural-Visual Affect Analysis in the Wild](https://mlsanity.com/paper/2002.03399/) (2020)
- [Factorized Higher-Order CNNs with an Application to Spatio-Temporal Emotion Estimation](https://mlsanity.com/paper/1906.06196/) (2019)
- [Bootstrapping Neural Processes](https://mlsanity.com/paper/2008.02956/) (2020)
- [Exploiting multi-CNN features in CNN-RNN based Dimensional Emotion Recognition on the OMG in-the-wild Dataset](https://mlsanity.com/paper/1910.01417/) (2019)
- [Multitask Emotion Recognition with Incomplete Labels](https://mlsanity.com/paper/2002.03557/) (2020)
- [A Multi-Task Learning & Generation Framework: Valence-Arousal, Action Units & Primary Expressions](https://mlsanity.com/paper/1811.07771/) (2018)
- [Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset](https://mlsanity.com/paper/1912.05008/) (2019)
- [Deep Structured Learning for Facial Action Unit Intensity Estimation](https://mlsanity.com/paper/1704.04481/) (2017)
- [The Ambiguous World of Emotion Representation](https://mlsanity.com/paper/1909.00360/) (2019)

## Built on by (most cited first, 16 of 16)

- [Benchmarking Micro-action Recognition: Dataset, Methods, and Applications](https://mlsanity.com/paper/2403.05234/) (2024)
- [COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2206.05833/) (2022)
- [MMAD: Multi-label Micro-Action Detection in Videos](https://mlsanity.com/paper/2407.05311/) (2024)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [FG-Net: Facial Action Unit Detection with Generalizable Pyramidal Features](https://mlsanity.com/paper/2308.12380/) (2023)
- [Optimal Transport-based Identity Matching for Identity-invariant Facial Expression Recognition](https://mlsanity.com/paper/2209.12172/) (2022)
- [Technical Report for Valence-Arousal Estimation in ABAW2 Challenge](https://mlsanity.com/paper/2107.03891/) (2021)
- [Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?](https://mlsanity.com/paper/2207.01113/) (2022)
- [In the Blink of an Eye: Event-based Emotion Recognition](https://mlsanity.com/paper/2310.04043/) (2023)
- [Apprenticeship-Inspired Elegance: Synergistic Knowledge Distillation Empowers Spiking Neural Networks for Efficient Single-Eye Emotion Recognition](https://mlsanity.com/paper/2407.09521/) (2024)
- [SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video](https://mlsanity.com/paper/2210.11341/) (2022)
- [AUEditNet: Dual-Branch Facial Action Unit Intensity Manipulation with Implicit Disentanglement](https://mlsanity.com/paper/2404.05063/) (2024)
- [Contextual Intelligent Decisions: Expert Moderation of Machine Outputs for Fair Assessment of Commercial Driving](https://mlsanity.com/paper/2202.09816/) (2022)
- [Occlusion Aware Student Emotion Recognition based on Facial Action Unit Detection](https://mlsanity.com/paper/2307.09465/) (2023)
- [xTrace: A Facial Expressive Behaviour Analysis Tool for Continuous Affect Recognition](https://mlsanity.com/paper/2505.05043/) (2025)
- [DIME: Scaling Facial Representation Learning via Differential Masked Autoencoding](https://mlsanity.com/paper/2610.17015/) (2026)

## Abstract (arXiv)

Temporal context is key to the recognition of expressions of emotion. Existing methods, that rely on recurrent or self-attention models to enforce temporal consistency, work on the feature level, ignoring the task-specific temporal dependencies, and fail to model context uncertainty. To alleviate these issues, we build upon the framework of Neural Processes to propose a method for apparent emotion recognition with three key novel components: (a) probabilistic contextual representation with a global latent variable model; (b) temporal context modelling using task-specific predictions in addition to features; and (c) smart temporal context selection. We validate our approach on four databases, two for Valence and Arousal estimation (SEWA and AffWild2), and two for Action Unit intensity estimation (DISFA and BP4D). Results show a consistent improvement over a series of strong baselines as well as over state-of-the-art methods.