# Facial Action Unit Detection Using Attention and Relation Learning

arXiv: https://arxiv.org/abs/1808.03457 · HTML: https://mlsanity.com/paper/1808.03457/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2018.

## Built on by (most cited first, 30 of 46)

- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2303.05416/) (2023)
- [Facial Affective Behavior Analysis with Instruction Tuning](https://mlsanity.com/paper/2404.05052/) (2024)
- [GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing](https://mlsanity.com/paper/2403.05916/) (2024)
- [LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis](https://mlsanity.com/paper/2308.10713/) (2023)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [Multi-scale Promoted Self-adjusting Correlation Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2308.07770/) (2023)
- [GRATIS: Deep Learning Graph Representation with Task-specific Topology and Multi-dimensional Edge Features](https://mlsanity.com/paper/2211.12482/) (2022)
- [Multi-label Transformer for Action Unit Detection](https://mlsanity.com/paper/2203.12531/) (2022)
- [Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning](https://mlsanity.com/paper/2408.00644/) (2024)
- [Spatio-Temporal AU Relational Graph Representation Learning For Facial Action Units Detection](https://mlsanity.com/paper/2303.10644/) (2023)
- [Meta Auxiliary Learning for Facial Action Unit Detection](https://mlsanity.com/paper/2105.06620/) (2021)
- [Multi-Order Networks for Action Unit Detection](https://mlsanity.com/paper/2202.00446/) (2022)
- [MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection](https://mlsanity.com/paper/2204.01349/) (2022)
- [Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample](https://mlsanity.com/paper/2410.01251/) (2024)
- [AFFDEX 2.0: A Real-Time Facial Expression Analysis Toolkit](https://mlsanity.com/paper/2202.12059/) (2022)
- [SS-MFAR : Semi-supervised Multi-task Facial Affect Recognition](https://mlsanity.com/paper/2207.09012/) (2022)
- [Contrastive Learning of Person-independent Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2403.03400/) (2024)
- [Automated Detection of Equine Facial Action Units](https://mlsanity.com/paper/2102.08983/) (2021)
- [An Attention-based Method for Action Unit Detection at the 3rd ABAW Competition](https://mlsanity.com/paper/2203.12428/) (2022)
- [Your "Attention" Deserves Attention: A Self-Diversified Multi-Channel Attention for Facial Action Analysis](https://mlsanity.com/paper/2203.12570/) (2022)
- [Multi-Level Adaptive Region of Interest and Graph Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/2102.12154/) (2021)
- [CausalAffect: Causal Discovery for Facial Affective Understanding](https://mlsanity.com/paper/2512.00456/) (2025)
- [Hierarchical Vision-Language Interaction for Facial Action Unit Detection](https://mlsanity.com/paper/2602.14425/) (2026)
- [Weakly Supervised Learning for Facial Affective Behavior Analysis : A Review](https://mlsanity.com/paper/2101.09858/) (2021)
- [Learning Contrastive Feature Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2402.06165/) (2024)
- [Automatic Facial Paralysis Estimation with Facial Action Units](https://mlsanity.com/paper/2203.01800/) (2022)
- [Knowledge-Spreader: Learning Facial Action Unit Dynamics with Extremely Limited Labels](https://mlsanity.com/paper/2203.16678/) (2022)
- [Facial Action Units Detection Aided by Global-Local Expression Embedding](https://mlsanity.com/paper/2210.13718/) (2022)

## Abstract (arXiv)

Attention mechanism has recently attracted increasing attentions in the field of facial action unit (AU) detection. By finding the region of interest of each AU with the attention mechanism, AU-related local features can be captured. Most of the existing attention based AU detection works use prior knowledge to predefine fixed attentions or refine the predefined attentions within a small range, which limits their capacity to model various AUs. In this paper, we propose an end-to-end deep learning based attention and relation learning framework for AU detection with only AU labels, which has not been explored before. In particular, multi-scale features shared by each AU are learned firstly, and then both channel-wise and spatial attentions are adaptively learned to select and extract AU-related local features. Moreover, pixel-level relations for AUs are further captured to refine spatial attentions so as to extract more relevant local features. Without changing the network architecture, our framework can be easily extended for AU intensity estimation. Extensive experiments show that our framework (i) soundly outperforms the state-of-the-art methods for both AU detection and AU intensity estimation on the challenging BP4D, DISFA, FERA 2015 and BP4D+ benchmarks, (ii) can adaptively capture the correlated regions of each AU, and (iii) also works well under severe occlusions and large poses.