# JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention

arXiv: https://arxiv.org/abs/2003.08834 · HTML: https://mlsanity.com/paper/2003.08834/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Mar 2020.

## Built on by (most cited first, 30 of 62)

- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectors](https://mlsanity.com/paper/2403.04697/) (2024)
- [EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning](https://mlsanity.com/paper/2408.11424/) (2024)
- [Facial Affective Behavior Analysis with Instruction Tuning](https://mlsanity.com/paper/2404.05052/) (2024)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis](https://mlsanity.com/paper/2308.10713/) (2023)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [Robust Light-Weight Facial Affective Behavior Recognition with CLIP](https://mlsanity.com/paper/2403.09915/) (2024)
- [Multi-scale Promoted Self-adjusting Correlation Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2308.07770/) (2023)
- [BigSmall: Efficient Multi-Task Learning for Disparate Spatial and Temporal Physiological Measurements](https://mlsanity.com/paper/2303.11573/) (2023)
- [GRATIS: Deep Learning Graph Representation with Task-specific Topology and Multi-dimensional Edge Features](https://mlsanity.com/paper/2211.12482/) (2022)
- [Self-supervised Facial Action Unit Detection with Region and Relation Learning](https://mlsanity.com/paper/2303.05708/) (2023)
- [FG-Net: Facial Action Unit Detection with Generalizable Pyramidal Features](https://mlsanity.com/paper/2308.12380/) (2023)
- [MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution](https://mlsanity.com/paper/2506.14511/) (2025)
- [Multi-scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition](https://mlsanity.com/paper/2404.06443/) (2024)
- [Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning](https://mlsanity.com/paper/2504.07198/) (2025)
- [Multi-label Transformer for Action Unit Detection](https://mlsanity.com/paper/2203.12531/) (2022)
- [Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning](https://mlsanity.com/paper/2408.00644/) (2024)
- [Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild](https://mlsanity.com/paper/2303.10590/) (2023)
- [Multi-Order Networks for Action Unit Detection](https://mlsanity.com/paper/2202.00446/) (2022)
- [MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection](https://mlsanity.com/paper/2204.01349/) (2022)
- [Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample](https://mlsanity.com/paper/2410.01251/) (2024)
- [FAN-Trans: Online Knowledge Distillation for Facial Action Unit Detection](https://mlsanity.com/paper/2211.06143/) (2022)
- [Local Region Perception and Relationship Learning Combined with Feature Fusion for Facial Action Unit Detection](https://mlsanity.com/paper/2303.08545/) (2023)
- [AU-TTT: Vision Test-Time Training model for Facial Action Unit Detection](https://mlsanity.com/paper/2503.23450/) (2025)
- [Your "Attention" Deserves Attention: A Self-Diversified Multi-Channel Attention for Facial Action Analysis](https://mlsanity.com/paper/2203.12570/) (2022)
- [Privileged Attribution Constrained Deep Networks for Facial Expression Recognition](https://mlsanity.com/paper/2203.12905/) (2022)
- [Using Positive Matching Contrastive Loss with Facial Action Units to mitigate bias in Facial Expression Recognition](https://mlsanity.com/paper/2303.04896/) (2023)
- [Vision Transformer for Action Units Detection](https://mlsanity.com/paper/2303.09917/) (2023)

## Abstract (arXiv)

Facial action unit (AU) detection and face alignment are two highly correlated tasks, since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. However, most existing AU detection works handle the two tasks independently by treating face alignment as a preprocessing, and often use landmarks to predefine a fixed region or attention for each AU. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared feature is learned firstly, and high-level feature of face alignment is fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment feature and global feature for AU detection. Extensive experiments demonstrate that our framework (i) significantly outperforms the state-of-the-art AU detection methods on the challenging BP4D, DISFA, GFT and BP4D+ benchmarks, (ii) can adaptively capture the irregular region of each AU, (iii) achieves competitive performance for face alignment, and (iv) also works well under partial occlusions and non-frontal poses. The code for our method is available at https://github.com/ZhiwenShao/PyTorch-JAANet.