# Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

arXiv: https://arxiv.org/abs/2303.10590 · HTML: https://mlsanity.com/paper/2303.10590/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 19 Mar 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network](https://mlsanity.com/paper/1910.11111/) (2019)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [Generative Hierarchical Features from Synthesizing Images](https://mlsanity.com/paper/2007.10379/) (2020)
- [Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition](https://mlsanity.com/paper/2205.01782/) (2022)
- [JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention](https://mlsanity.com/paper/2003.08834/) (2020)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [Facial Action Unit Recognition With Multi-models Ensembling](https://mlsanity.com/paper/2203.13046/) (2022)

## Built on by (most cited first, 8 of 8)

- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [Affective Behaviour Analysis via Integrating Multi-Modal Knowledge](https://mlsanity.com/paper/2403.10825/) (2024)
- [Affective Behaviour Analysis via Progressive Learning](https://mlsanity.com/paper/2407.16945/) (2024)
- [AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and GPT-2 in Wild Audiovisual Contexts](https://mlsanity.com/paper/2403.13678/) (2024)
- [Exploring CausalWorld: Enhancing robotic manipulation via knowledge transfer and curriculum learning](https://mlsanity.com/paper/2403.17266/) (2024)
- [AU-vMAE: Knowledge-Guide Action Units Detection via Video Masked Autoencoder](https://mlsanity.com/paper/2407.11468/) (2024)
- [7ABAW-Compound Expression Recognition via Curriculum Learning](https://mlsanity.com/paper/2503.07969/) (2025)

## Abstract (arXiv)

Facial action unit detection has emerged as an important task within facial expression analysis, aimed at detecting specific pre-defined, objective facial expressions, such as lip tightening and cheek raising. This paper presents our submission to the Affective Behavior Analysis in-the-wild (ABAW) 2023 Competition for AU detection. We propose a multi-modal method for facial action unit detection with visual, acoustic, and lexical features extracted from the large pre-trained models. To provide high-quality details for visual feature extraction, we apply super-resolution and face alignment to the training data and show potential performance gain. Our approach achieves the F1 score of 52.3% on the official validation set of the 5th ABAW Challenge.