# Deep Learning for Human Affect Recognition: Insights and New Developments

arXiv: https://arxiv.org/abs/1901.02884 · HTML: https://mlsanity.com/paper/1901.02884/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2019.

## Built on by (most cited first, 23 of 23)

- [Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2103.09154/) (2021)
- [Multimodal Engagement Analysis from Facial Videos in the Classroom](https://mlsanity.com/paper/2101.04215/) (2021)
- [Fine-grained Affective Processing Capabilities Emerging from Large Language Models](https://mlsanity.com/paper/2309.01664/) (2023)
- [COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition](https://mlsanity.com/paper/2206.05833/) (2022)
- [SVFAP: Self-supervised Video Facial Affect Perceiver](https://mlsanity.com/paper/2401.00416/) (2023)
- [Graph-based Facial Affect Analysis: A Review](https://mlsanity.com/paper/2103.15599/) (2021)
- [High-Level Context Representation for Emotion Recognition in Images](https://mlsanity.com/paper/2305.03500/) (2023)
- [RegBN: Batch Normalization of Multimodal Data with Regularization](https://mlsanity.com/paper/2310.00641/) (2023)
- [Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer](https://mlsanity.com/paper/2404.17205/) (2024)
- [Gaze-enhanced Crossmodal Embeddings for Emotion Recognition](https://mlsanity.com/paper/2205.00129/) (2022)
- [AU-Aware Vision Transformers for Biased Facial Expression Recognition](https://mlsanity.com/paper/2211.06609/) (2022)
- [Towards A Robust Group-level Emotion Recognition via Uncertainty-Aware Learning](https://mlsanity.com/paper/2310.04306/) (2023)
- [Exploring Emotion Expression Recognition in Older Adults Interacting with a Virtual Coach](https://mlsanity.com/paper/2311.05567/) (2023)
- [MAFER: a Multi-resolution Approach to Facial Expression Recognition](https://mlsanity.com/paper/2105.02481/) (2021)
- [HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer](https://mlsanity.com/paper/2305.12369/) (2023)
- [Nonverbal Immediacy Analysis in Education: A Multimodal Computational Model](https://mlsanity.com/paper/2407.17209/) (2024)
- [Expression Recognition Analysis in the Wild](https://mlsanity.com/paper/2101.09231/) (2021)
- [A Multi-resolution Approach to Expression Recognition in the Wild](https://mlsanity.com/paper/2103.05723/) (2021)
- [Multi-Cue Adaptive Emotion Recognition Network](https://mlsanity.com/paper/2111.02273/) (2021)
- [AU-Supervised Convolutional Vision Transformers for Synthetic Facial Expression Recognition](https://mlsanity.com/paper/2207.09777/) (2022)
- [Teleology-Driven Affective Computing: A Causal Framework for Sustained Well-Being](https://mlsanity.com/paper/2502.17172/) (2025)
- [Learning Transferable Facial Emotion Representations from Large-Scale Semantically Rich Captions](https://mlsanity.com/paper/2507.21015/) (2025)
- [LaScA: Language-Conditioned Scalable Modelling of Affective Dynamics](https://mlsanity.com/paper/2604.07193/) (2026)

## Abstract (arXiv)

Automatic human affect recognition is a key step towards more natural human-computer interaction. Recent trends include recognition in the wild using a fusion of audiovisual and physiological sensors, a challenging setting for conventional machine learning algorithms. Since 2010, novel deep learning algorithms have been applied increasingly in this field. In this paper, we review the literature on human affect recognition between 2010 and 2017, with a special focus on approaches using deep neural networks. By classifying a total of 950 studies according to their usage of shallow or deep architectures, we are able to show a trend towards deep learning. Reviewing a subset of 233 studies that employ deep neural networks, we comprehensively quantify their applications in this field. We find that deep learning is used for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal sensor data. Exemplary state-of-the-art architectures illustrate the progress. Our findings show the role deep architectures will play in human affect recognition, and can serve as a reference point for researchers working on related applications.