# Spatio-Temporal AU Relational Graph Representation Learning For Facial Action Units Detection

arXiv: https://arxiv.org/abs/2303.10644 · HTML: https://mlsanity.com/paper/2303.10644/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 60 references that are in the map.
- First posted on arXiv 19 Mar 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network](https://mlsanity.com/paper/1910.11111/) (2019)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/1904.09939/) (2019)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [Facial Action Unit Detection Using Attention and Relation Learning](https://mlsanity.com/paper/1808.03457/) (2018)
- [Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusing](https://mlsanity.com/paper/1704.03067/) (2017)
- [EAC-Net: A Region-based Deep Enhancing and Cropping Approach for Facial Action Unit Detection](https://mlsanity.com/paper/1702.02925/) (2017)
- [A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition](https://mlsanity.com/paper/2107.04187/) (2021)
- [Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition](https://mlsanity.com/paper/2107.03708/) (2021)
- [Facial Action Unit Detection and Intensity Estimation from Self-supervised Representation](https://mlsanity.com/paper/2210.15878/) (2022)
- [GRATIS: Deep Learning Graph Representation with Task-specific Topology and Multi-dimensional Edge Features](https://mlsanity.com/paper/2211.12482/) (2022)
- [Affective Behavior Analysis using Action Unit Relation Graph and Multi-task Cross Attention](https://mlsanity.com/paper/2207.10293/) (2022)
- [A transformer-based approach to video frame-level prediction in Affective Behaviour Analysis In-the-wild](https://mlsanity.com/paper/2303.09293/) (2023)
- [EmotiEffNet Facial Features in Uni-task Emotion Recognition in Video at ABAW-5 competition](https://mlsanity.com/paper/2303.09162/) (2023)

## Built on by (most cited first, 7 of 7)

- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [Multi-scale Dynamic and Hierarchical Relationship Modeling for Facial Action Units Recognition](https://mlsanity.com/paper/2404.06443/) (2024)
- [Emotion Recognition Using Transformers with Masked Learning](https://mlsanity.com/paper/2403.13731/) (2024)
- [AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and GPT-2 in Wild Audiovisual Contexts](https://mlsanity.com/paper/2403.13678/) (2024)
- [AU-vMAE: Knowledge-Guide Action Units Detection via Video Masked Autoencoder](https://mlsanity.com/paper/2407.11468/) (2024)

## Abstract (arXiv)

This paper presents our Facial Action Units (AUs) detection submission to the fifth Affective Behavior Analysis in-the-wild Competition (ABAW). Our approach consists of three main modules: (i) a pre-trained facial representation encoder which produce a strong facial representation from each input face image in the input sequence; (ii) an AU-specific feature generator that specifically learns a set of AU features from each facial representation; and (iii) a spatio-temporal graph learning module that constructs a spatio-temporal graph representation. This graph representation describes AUs contained in all frames and predicts the occurrence of each AU based on both the modeled spatial information within the corresponding face and the learned temporal dynamics among frames. The experimental results show that our approach outperformed the baseline and the spatio-temporal graph representation learning allows our model to generate the best results among all ablated systems. Our model ranks at the 4th place in the AU recognition track at the 5th ABAW Competition. Our code is publicly available at https://github.com/wzh125/ABAW-5.