# Multi-scale Promoted Self-adjusting Correlation Learning for Facial Action Unit Detection

arXiv: https://arxiv.org/abs/2308.07770 · HTML: https://mlsanity.com/paper/2308.07770/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 72 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [Residual Gated Graph ConvNets](https://mlsanity.com/paper/1711.07553/) (2017)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Vision GNN: An Image is Worth Graph of Nodes](https://mlsanity.com/paper/2206.00272/) (2022)
- [CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention](https://mlsanity.com/paper/2108.00154/) (2021)
- [MPViT: Multi-Path Vision Transformer for Dense Prediction](https://mlsanity.com/paper/2112.11010/) (2021)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [ResT: An Efficient Transformer for Visual Recognition](https://mlsanity.com/paper/2105.13677/) (2021)
- [iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis](https://mlsanity.com/paper/2107.00285/) (2021)
- [JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention](https://mlsanity.com/paper/2003.08834/) (2020)
- [Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment](https://mlsanity.com/paper/1803.05588/) (2018)
- [Semantic Relationships Guided Representation Learning for Facial Action Unit Recognition](https://mlsanity.com/paper/1904.09939/) (2019)
- [Facial Action Unit Detection Using Attention and Relation Learning](https://mlsanity.com/paper/1808.03457/) (2018)
- [Deep Structure Inference Network for Facial Action Unit Recognition](https://mlsanity.com/paper/1803.05873/) (2018)
- [Relation Modeling with Graph Convolutional Networks for Facial Action Unit Detection](https://mlsanity.com/paper/1910.10334/) (2019)
- [Graph-based Facial Affect Analysis: A Review](https://mlsanity.com/paper/2103.15599/) (2021)
- [Joint Action Unit localisation and intensity estimation through heatmap regression](https://mlsanity.com/paper/1805.03487/) (2018)
- [Uncertain Facial Expression Recognition via Multi-task Assisted Correction](https://mlsanity.com/paper/2212.07144/) (2022)
- [Optimizing Filter Size in Convolutional Neural Networks for Facial Action Unit Recognition](https://mlsanity.com/paper/1707.08630/) (2017)

## Built on by (most cited first, 14 of 14)

- [AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectors](https://mlsanity.com/paper/2403.04697/) (2024)
- [EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning](https://mlsanity.com/paper/2408.11424/) (2024)
- [A Simple yet Effective Network based on Vision Transformer for Camouflaged Object and Salient Object Detection](https://mlsanity.com/paper/2402.18922/) (2024)
- [GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing](https://mlsanity.com/paper/2403.05916/) (2024)
- [Representation Learning and Identity Adversarial Training for Facial Behavior Understanding](https://mlsanity.com/paper/2407.11243/) (2024)
- [MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution](https://mlsanity.com/paper/2506.14511/) (2025)
- [Language-Driven Interactive Shadow Detection](https://mlsanity.com/paper/2408.08543/) (2024)
- [AU-TTT: Vision Test-Time Training model for Facial Action Unit Detection](https://mlsanity.com/paper/2503.23450/) (2025)
- [Beyond FACS: Data-driven Facial Expression Dictionaries, with Application to Predicting Autism](https://mlsanity.com/paper/2505.24679/) (2025)
- [Hybrid-supervised Hypergraph-enhanced Transformer for Micro-gesture Based Emotion Recognition](https://mlsanity.com/paper/2507.14867/) (2025)
- [Distribution-Specific Learning for Joint Salient and Camouflaged Object Detection](https://mlsanity.com/paper/2508.06063/) (2025)
- [Learning Contrastive Feature Representations for Facial Action Unit Detection](https://mlsanity.com/paper/2402.06165/) (2024)
- [One-Frame Calibration with Siamese Network in Facial Action Unit Recognition](https://mlsanity.com/paper/2409.00240/) (2024)
- [Adversarial Robustness in RGB-Skeleton Action Recognition: Leveraging Attention Modality Reweighter](https://mlsanity.com/paper/2407.19981/) (2024)

## Abstract (arXiv)

Facial Action Unit (AU) detection is a crucial task in affective computing and social robotics as it helps to identify emotions expressed through facial expressions. Anatomically, there are innumerable correlations between AUs, which contain rich information and are vital for AU detection. Previous methods used fixed AU correlations based on expert experience or statistical rules on specific benchmarks, but it is challenging to comprehensively reflect complex correlations between AUs via hand-crafted settings. There are alternative methods that employ a fully connected graph to learn these dependencies exhaustively. However, these approaches can result in a computational explosion and high dependency with a large dataset. To address these challenges, this paper proposes a novel self-adjusting AU-correlation learning (SACL) method with less computation for AU detection. This method adaptively learns and updates AU correlation graphs by efficiently leveraging the characteristics of different levels of AU motion and emotion representation information extracted in different stages of the network. Moreover, this paper explores the role of multi-scale learning in correlation information extraction, and design a simple yet effective multi-scale feature learning (MSFL) method to promote better performance in AU detection. By integrating AU correlation information with multi-scale features, the proposed method obtains a more robust feature representation for the final AU detection. Extensive experiments show that the proposed method outperforms the state-of-the-art methods on widely used AU detection benchmark datasets, with only 28.7% and 12.0% of the parameters and FLOPs of the best method, respectively. The code for this method is available at urlhttps://github.com/linuxsino/Self-adjusting-AU.