mlsanityData to 14 Oct 2026

Multi-modal Facial Action Unit Detection with Large Pre-trained Models for the 5th Competition on Affective Behavior Analysis in-the-wild

arXiv 2303.10590 ↗First posted 19 Mar 2023cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

Facial action unit detection has emerged as an important task within facial expression analysis, aimed at detecting specific pre-defined, objective facial expressions, such as lip tightening and cheek raising. This paper presents our submission to the Affective Behavior Analysis in-the-wild (ABAW) 2023 Competition for AU detection. We propose a multi-modal method for facial action unit detection with visual, acoustic, and lexical features extracted from the large pre-trained models. To provide high-quality details for visual feature extraction, we apply super-resolution and face alignment to the training data and show potential performance gain. Our approach achieves the F1 score of 52.3% on the official validation set of the 5th ABAW Challenge.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 19 Mar 2023.

Builds on

25 of 36 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 970
  11. 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
  12. 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
13 more
  1. 2020Analysing Affective Behavior in the First ABAW 2020 Competitioncited by 178
  2. 2021Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Frameworkcited by 174
  3. 2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
  4. 2021Analysing Affective Behavior in the second ABAW2 Competitioncited by 163
  5. 2022ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challengescited by 156
  6. 2019Face Behavior a la carte: Expressions, Affect and Action Units in a Single Networkcited by 155
  7. 2022ABAW: Learning from Synthetic Data & Multi-Task Learning Challengescited by 110
  8. 2023ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challengescited by 110
  9. 2020Generative Hierarchical Features from Synthesizing Imagescited by 83
  10. 2022Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognitioncited by 82
  11. 2020JhatAA-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attentioncited by 62
  12. 2022Transformer-based Multimodal Information Fusion for Facial Expression Analysiscited by 49
  13. 2022Facial Action Unit Recognition With Multi-models Ensemblingcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 7; 2025: 1.

8 papers.

  1. 2024The 6th Affective Behavior Analysis in-the-wild (ABAW) Competitioncited by 78
  2. 20247th ABAW Competition: Multi-Task Learning and Compound Expression Recognitioncited by 53
  3. 2024Affective Behaviour Analysis via Integrating Multi-Modal Knowledgecited by 18
  4. 2024Affective Behaviour Analysis via Progressive Learningcited by 7
  5. 2024AUD-TGN: Advancing Action Unit Detection with Temporal Convolution and GPT-2 in Wild Audiovisual Contextscited by 3
  6. 2024Exploring CausalWorld: Enhancing robotic manipulation via knowledge transfer and curriculum learningcited by 2
  7. 2024AU-vMAE: Knowledge-Guide Action Units Detection via Video Masked Autoencodercited by 2
  8. 20257ABAW-Compound Expression Recognition via Curriculum Learningcited by 0