# Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample

arXiv: https://arxiv.org/abs/2410.01251 · HTML: https://mlsanity.com/paper/2410.01251/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 64 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [Interventional Few-Shot Learning](https://mlsanity.com/paper/2009.13000/) (2020)
- [Visual Commonsense R-CNN](https://mlsanity.com/paper/2002.12204/) (2020)
- [Discovering Causal Signals in Images](https://mlsanity.com/paper/1605.08179/) (2016)
- [ResT: An Efficient Transformer for Visual Recognition](https://mlsanity.com/paper/2105.13677/) (2021)
- [Two Causal Principles for Improving Visual Dialog](https://mlsanity.com/paper/1911.10496/) (2019)
- [JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention](https://mlsanity.com/paper/2003.08834/) (2020)
- [Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/1904.09939/) (2019)
- [Facial Action Unit Detection Using Attention and Relation Learning](https://mlsanity.com/paper/1808.03457/) (2018)
- [Deep Structure Inference Network for Facial Action Unit Recognition](https://mlsanity.com/paper/1803.05873/) (2018)
- [Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection](https://mlsanity.com/paper/1910.10334/) (2019)
- [AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection](https://mlsanity.com/paper/1812.05788/) (2018)
- [Causal Intervention for Subject-Deconfounded Facial Action Unit Recognition](https://mlsanity.com/paper/2204.07935/) (2022)
- [Shape Constrained Network for Eye Segmentation in the Wild](https://mlsanity.com/paper/1910.05283/) (2019)
- [ResT V2: Simpler, Faster and Stronger](https://mlsanity.com/paper/2204.07366/) (2022)
- [Facial Action Unit Detection via Adaptive Attention and Relation](https://mlsanity.com/paper/2001.01168/) (2020)

## Built on by (most cited first, 6 of 6)

- [MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution](https://mlsanity.com/paper/2506.14511/) (2025)
- [Hybrid Generative Fusion for Efficient and Privacy-Preserving Face Recognition Dataset Generation](https://mlsanity.com/paper/2508.10672/) (2025)
- [Micro-Expression Recognition via Fine-Grained Dynamic Perception](https://mlsanity.com/paper/2509.06015/) (2025)
- [Symmetric Perception and Ordinal Regression for Detecting Scoliosis Natural Image](https://mlsanity.com/paper/2411.15799/) (2024)
- [A Benchmark for Incremental Micro-expression Recognition](https://mlsanity.com/paper/2501.19111/) (2025)
- [UAU-Net: Uncertainty-aware Representation Learning and Evidential Classification for Facial Action Unit Detection](https://mlsanity.com/paper/2604.21227/) (2026)

## Abstract (arXiv)

Facial action unit (AU) detection remains a challenging task, due to the subtlety, dynamics, and diversity of AUs. Recently, the prevailing techniques of self-attention and causal inference have been introduced to AU detection. However, most existing methods directly learn self-attention guided by AU detection, or employ common patterns for all AUs during causal intervention. The former often captures irrelevant information in a global range, and the latter ignores the specific causal characteristic of each AU. In this paper, we propose a novel AU detection framework called AC2D by adaptively constraining self-attention weight distribution and causally deconfounding the sample confounder. Specifically, we explore the mechanism of self-attention weight distribution, in which the self-attention weight distribution of each AU is regarded as spatial distribution and is adaptively learned under the constraint of location-predefined attention and the guidance of AU detection. Moreover, we propose a causal intervention module for each AU, in which the bias caused by training samples and the interference from irrelevant AUs are both suppressed. Extensive experiments show that our method achieves competitive performance compared to state-of-the-art AU detection approaches on challenging benchmarks, including BP4D, DISFA, GFT, and BP4D+ in constrained scenarios and Aff-Wild2 in unconstrained scenarios. The code is available at https://github.com/ZhiwenShao/AC2D.