# Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment

arXiv: https://arxiv.org/abs/1803.05588 · HTML: https://mlsanity.com/paper/1803.05588/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2018.

## Built on by (most cited first, 30 of 55)

- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing](https://mlsanity.com/paper/2403.05916/) (2024)
- [Domain-Incremental Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition](https://mlsanity.com/paper/2103.08637/) (2021)
- [LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis](https://mlsanity.com/paper/2308.10713/) (2023)
- [Pose-disentangled Contrastive Learning for Self-supervised Facial Representation](https://mlsanity.com/paper/2211.13490/) (2022)
- [Weakly-Supervised Text-driven Contrastive Learning for Facial Behavior Understanding](https://mlsanity.com/paper/2304.00058/) (2023)
- [Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit](https://mlsanity.com/paper/2409.17717/) (2024)
- [Multi-scale Promoted Self-adjusting Correlation Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2308.07770/) (2023)
- [BigSmall: Efficient Multi-Task Learning for Disparate Spatial and Temporal Physiological Measurements](https://mlsanity.com/paper/2303.11573/) (2023)
- [Talking Head Generation with Audio and Speech Related Facial Action Units](https://mlsanity.com/paper/2110.09951/) (2021)
- [Causal Intervention for Subject-Deconfounded Facial Action Unit Recognition](https://mlsanity.com/paper/2204.07935/) (2022)
- [GRATIS: Deep Learning Graph Representation with Task-specific Topology and Multi-dimensional Edge Features](https://mlsanity.com/paper/2211.12482/) (2022)
- [Self-supervised Facial Action Unit Detection with Region and Relation Learning](https://mlsanity.com/paper/2303.05708/) (2023)
- [VEATIC: Video-based Emotion and Affect Tracking in Context Dataset](https://mlsanity.com/paper/2309.06745/) (2023)
- [Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection](https://mlsanity.com/paper/2209.12244/) (2022)
- [Multi-label Transformer for Action Unit Detection](https://mlsanity.com/paper/2203.12531/) (2022)
- [Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning](https://mlsanity.com/paper/2408.00644/) (2024)
- [Meta Auxiliary Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2105.06620/) (2021)
- [Multi-Order Networks for Action Unit Detection](https://mlsanity.com/paper/2202.00446/) (2022)
- [MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection](https://mlsanity.com/paper/2204.01349/) (2022)
- [AFFDEX 2.0: A Real-Time Facial Expression Analysis Toolkit](https://mlsanity.com/paper/2202.12059/) (2022)
- [Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers](https://mlsanity.com/paper/2203.11441/) (2022)
- [FAN-Trans: Online Knowledge Distillation for Facial Action Unit Detection](https://mlsanity.com/paper/2211.06143/) (2022)
- [Contrastive Learning of Person-independent Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2403.03400/) (2024)
- [Your "Attention" Deserves Attention: A Self-Diversified Multi-Channel Attention for Facial Action Analysis](https://mlsanity.com/paper/2203.12570/) (2022)
- [FreeAvatar: Robust 3D Facial Animation Transfer by Learning an Expression Foundation Model](https://mlsanity.com/paper/2409.13180/) (2024)
- [Multi-Level Adaptive Region of Interest and Graph Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/2102.12154/) (2021)
- [Towards Fair Affective Robotics: Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition](https://mlsanity.com/paper/2103.09233/) (2021)
- [Feature Pyramid Network for Multi-task Affective Analysis](https://mlsanity.com/paper/2107.03670/) (2021)

## Abstract (arXiv)

Facial action unit (AU) detection and face alignment are two highly correlated tasks since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. Most existing AU detection works often treat face alignment as a preprocessing and handle the two tasks independently. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared features are learned firstly, and high-level features of face alignment are fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment features and global features for AU detection. Experiments on BP4D and DISFA benchmarks demonstrate that our framework significantly outperforms the state-of-the-art methods for AU detection.