# DeepCoder: Semi-parametric Variational Autoencoders for Automatic Facial Action Coding

arXiv: https://arxiv.org/abs/1704.02206 · HTML: https://mlsanity.com/paper/1704.02206/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2017.

## Built on by (most cited first, 9 of 9)

- [Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition](https://mlsanity.com/paper/2103.13372/) (2021)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [FG-Net: Facial Action Unit Detection with Generalizable Pyramidal Features](https://mlsanity.com/paper/2308.12380/) (2023)
- [Are 3D Face Shapes Expressive Enough for Recognising Continuous Emotions and Action Unit Intensities?](https://mlsanity.com/paper/2207.01113/) (2022)
- [One-Frame Calibration with Siamese Network in Facial Action Unit Recognition](https://mlsanity.com/paper/2409.00240/) (2024)
- [A Shared Latent for Partially-Labeled Multi-Task Facial Affect Recognition](https://mlsanity.com/paper/2607.16285/) (2026)
- [FERGI: Automatic Scoring of User Preferences for Text-to-Image Generation from Spontaneous Facial Expression Reaction](https://mlsanity.com/paper/2312.03187/) (2023)
- [Trend-Aware Supervision: On Learning Invariance for Semi-Supervised Facial Action Unit Intensity Estimation](https://mlsanity.com/paper/2503.08078/) (2025)
- [Facial-Expression-Aware Prompting for Empathetic LLM Tutoring](https://mlsanity.com/paper/2604.15336/) (2026)

## Abstract (arXiv)

Human face exhibits an inherent hierarchy in its representations (i.e., holistic facial expressions can be encoded via a set of facial action units (AUs) and their intensity). Variational (deep) auto-encoders (VAE) have shown great results in unsupervised extraction of hierarchical latent representations from large amounts of image data, while being robust to noise and other undesired artifacts. Potentially, this makes VAEs a suitable approach for learning facial features for AU intensity estimation. Yet, most existing VAE-based methods apply classifiers learned separately from the encoded features. By contrast, the non-parametric (probabilistic) approaches, such as Gaussian Processes (GPs), typically outperform their parametric counterparts, but cannot deal easily with large amounts of data. To this end, we propose a novel VAE semi-parametric modeling framework, named DeepCoder, which combines the modeling power of parametric (convolutional) and nonparametric (ordinal GPs) VAEs, for joint learning of (1) latent representations at multiple levels in a task hierarchy1, and (2) classification of multiple ordinal outputs. We show on benchmark datasets for AU intensity estimation that the proposed DeepCoder outperforms the state-of-the-art approaches, and related VAEs and deep learning models.