# Deep Structured Learning for Facial Action Unit Intensity Estimation

arXiv: https://arxiv.org/abs/1704.04481 · HTML: https://mlsanity.com/paper/1704.04481/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Apr 2017.

## Built on by (most cited first, 17 of 17)

- [Affective Processes: stochastic modelling of temporal context for emotion and facial expression recognition](https://mlsanity.com/paper/2103.13372/) (2021)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [FG-Net: Facial Action Unit Detection with Generalizable Pyramidal Features](https://mlsanity.com/paper/2308.12380/) (2023)
- [Personalized Federated Deep Learning for Pain Estimation From Face Images](https://mlsanity.com/paper/2101.04800/) (2021)
- [Using Self-Supervised Auxiliary Tasks to Improve Fine-Grained Facial Representation](https://mlsanity.com/paper/2105.06421/) (2021)
- [Go-Blend behavior and affect](https://mlsanity.com/paper/2109.13388/) (2021)
- [Play with Emotion: Affect-Driven Reinforcement Learning](https://mlsanity.com/paper/2208.12622/) (2022)
- [FAN-Trans: Online Knowledge Distillation for Facial Action Unit Detection](https://mlsanity.com/paper/2211.06143/) (2022)
- [Multi-Level Adaptive Region of Interest and Graph Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/2102.12154/) (2021)
- [CeCNN: Copula-enhanced convolutional neural networks in joint prediction of refraction error and axial length based on ultra-widefield fundus images](https://mlsanity.com/paper/2311.03967/) (2023)
- [One-Frame Calibration with Siamese Network in Facial Action Unit Recognition](https://mlsanity.com/paper/2409.00240/) (2024)
- [Hierarchical Granularity Alignment and State Space Modeling for Robust Multimodal AU Detection in the Wild](https://mlsanity.com/paper/2603.11306/) (2026)
- [FERGI: Automatic Scoring of User Preferences for Text-to-Image Generation from Spontaneous Facial Expression Reaction](https://mlsanity.com/paper/2312.03187/) (2023)
- [Trend-Aware Supervision: On Learning Invariance for Semi-Supervised Facial Action Unit Intensity Estimation](https://mlsanity.com/paper/2503.08078/) (2025)
- [Solution for 8th Competition on Affective & Behavior Analysis in-the-wild](https://mlsanity.com/paper/2503.11115/) (2025)
- [Facial-Expression-Aware Prompting for Empathetic LLM Tutoring](https://mlsanity.com/paper/2604.15336/) (2026)
- [Capturing Dynamics: The 4D Facial Expression Intensity Dataset](https://mlsanity.com/paper/2610.02647/) (2026)

## Abstract (arXiv)

We consider the task of automated estimation of facial expression intensity. This involves estimation of multiple output variables (facial action units --- AUs) that are structurally dependent. Their structure arises from statistically induced co-occurrence patterns of AU intensity levels. Modeling this structure is critical for improving the estimation performance; however, this performance is bounded by the quality of the input features extracted from face images. The goal of this paper is to model these structures and estimate complex feature representations simultaneously by combining conditional random field (CRF) encoded AU dependencies with deep learning. To this end, we propose a novel Copula CNN deep learning approach for modeling multivariate ordinal variables. Our model accounts for ordinal structure in output variables and their non-linear dependencies via copula functions modeled as cliques of a CRF. These are jointly optimized with deep CNN feature encoding layers using a newly introduced balanced batch iterative training algorithm. We demonstrate the effectiveness of our approach on the task of AU intensity estimation on two benchmark datasets. We show that joint learning of the deep features and the target output structure results in significant performance gains compared to existing deep structured models for analysis of facial expressions.