# Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusing

arXiv: https://arxiv.org/abs/1704.03067 · HTML: https://mlsanity.com/paper/1704.03067/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2017.

## Built on by (most cited first, 28 of 28)

- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Deep Learning for Micro-expression Recognition: A Survey](https://mlsanity.com/paper/2107.02823/) (2021)
- [Causal Intervention for Subject-Deconfounded Facial Action Unit Recognition](https://mlsanity.com/paper/2204.07935/) (2022)
- [Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning](https://mlsanity.com/paper/2408.00644/) (2024)
- [Spatio-Temporal AU Relational Graph Representation Learning For Facial Action Units Detection](https://mlsanity.com/paper/2303.10644/) (2023)
- [Unobtrusive Pain Monitoring in Older Adults with Dementia using Pairwise and Contrastive Training](https://mlsanity.com/paper/2101.03251/) (2021)
- [Meta Auxiliary Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2105.06620/) (2021)
- [Action Units Recognition Using Improved Pairwise Deep Architecture](https://mlsanity.com/paper/2107.03143/) (2021)
- [MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection](https://mlsanity.com/paper/2204.01349/) (2022)
- [Contrastive Learning of Person-independent Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2403.03400/) (2024)
- [An Attention-based Method for Action Unit Detection at the 3rd ABAW Competition](https://mlsanity.com/paper/2203.12428/) (2022)
- [Multi-Level Adaptive Region of Interest and Graph Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/2102.12154/) (2021)
- [Weakly Supervised Regional and Temporal Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/2204.00379/) (2022)
- [Hierarchical Vision-Language Interaction for Facial Action Unit Detection](https://mlsanity.com/paper/2602.14425/) (2026)
- [Learning Contrastive Feature Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2402.06165/) (2024)
- [Towards Unified Facial Action Unit Recognition Framework by Large Language Models](https://mlsanity.com/paper/2409.08444/) (2024)
- [Self-Supervised Regional and Temporal Auxiliary Tasks for Facial Action Unit Recognition](https://mlsanity.com/paper/2107.14399/) (2021)
- [Automatic Facial Paralysis Estimation with Facial Action Units](https://mlsanity.com/paper/2203.01800/) (2022)
- [Knowledge-Spreader: Learning Facial Action Unit Dynamics with Extremely Limited Labels](https://mlsanity.com/paper/2203.16678/) (2022)
- [Facial Action Units Detection Aided by Global-Local Expression Embedding](https://mlsanity.com/paper/2210.13718/) (2022)
- [Exploring Adversarial Learning for Deep Semi-Supervised Facial Action Unit Recognition](https://mlsanity.com/paper/2106.02258/) (2021)
- [EventFormer: AU Event Transformer for Facial Action Unit Event Detection](https://mlsanity.com/paper/2203.06355/) (2022)
- [Cross-subject Action Unit Detection with Meta Learning and Transformer-based Relation Modeling](https://mlsanity.com/paper/2205.08787/) (2022)
- [FERGI: Automatic Scoring of User Preferences for Text-to-Image Generation from Spontaneous Facial Expression Reaction](https://mlsanity.com/paper/2312.03187/) (2023)
- [Solution for 8th Competition on Affective & Behavior Analysis in-the-wild](https://mlsanity.com/paper/2503.11115/) (2025)
- [Saliency-guided Emotion Modeling: Predicting Viewer Reactions from Video Stimuli](https://mlsanity.com/paper/2505.19178/) (2025)
- [Facial-Expression-Aware Prompting for Empathetic LLM Tutoring](https://mlsanity.com/paper/2604.15336/) (2026)
- [MEDN: Motion-Emotion Feature Decoupling Network for Micro-Expression Recognition](https://mlsanity.com/paper/2604.17899/) (2026)

## Abstract (arXiv)

Action Unit (AU) detection becomes essential for facial analysis. Many proposed approaches face challenging problems in dealing with the alignments of different face regions, in the effective fusion of temporal information, and in training a model for multiple AU labels. To better address these problems, we propose a deep learning framework for AU detection with region of interest (ROI) adaptation, integrated multi-label learning, and optimal LSTM-based temporal fusing. First, ROI cropping nets (ROI Nets) are designed to make sure specifically interested regions of faces are learned independently; each sub-region has a local convolutional neural network (CNN) - an ROI Net, whose convolutional filters will only be trained for the corresponding region. Second, multi-label learning is employed to integrate the outputs of those individual ROI cropping nets, which learns the inter-relationships of various AUs and acquires global features across sub-regions for AU detection. Finally, the optimal selection of multiple LSTM layers to form the best LSTM Net is carried out to best fuse temporal features, in order to make the AU prediction the most accurate. The proposed approach is evaluated on two popular AU detection datasets, BP4D and DISFA, outperforming the state of the art significantly, with an average improvement of around 13% on BP4D and 25% on DISFA, respectively.