mlsanityData to 7 Oct 2026

A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition

arXiv 2107.04187 ↗First posted 9 Jul 2021cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

Analyzing human affect is vital for human-computer interaction systems. Most methods are developed in restricted scenarios which are not practical for in-the-wild settings. The Affective Behavior Analysis in-the-wild (ABAW) 2021 Contest provides a benchmark for this in-the-wild problem. In this paper, we introduce a multi-modal and multi-task learning method by using both visual and audio information. We use both AU and expression annotations to train the model and apply a sequence model to further extract associations between video frames. We achieve an AU score of 0.712 and an expression score of 0.477 on the validation set. These results demonstrate the effectiveness of our approach in improving model performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 17 of its 22 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

17 of 22 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2016An overview of gradient descent optimization algorithmscited by 1,021
  6. 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
  7. 2019Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFacecited by 222
  8. 2018Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyondcited by 216
  9. 2020Analysing Affective Behavior in the First ABAW 2020 Competitioncited by 178
  10. 2021Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Frameworkcited by 174
  11. 2021Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Studycited by 168
  12. 2020Partial FC: Training 10 Million Identities on a Single Machinecited by 163
5 more
  1. 2021Analysing Affective Behavior in the second ABAW2 Competitioncited by 163
  2. 2019Face Behavior a la carte: Expressions, Affect and Action Units in a Single Networkcited by 155
  3. 2018Aff-Wild2: Extending the Aff-Wild Database for Affect Recognitioncited by 80
  4. 2020Two-Stream Aural-Visual Affect Analysis in the Wildcited by 51
  5. 2020Multitask Emotion Recognition with Incomplete Labelscited by 39

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 10; 2023: 6; 2024: 4; 2025: 3.

24 papers.

Sort:

  1. 2022ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challengescited by 156
  2. 2022ABAW: Learning from Synthetic Data & Multi-Task Learning Challengescited by 110
  3. 2023ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challengescited by 110
  4. 2024The 6th Affective Behavior Analysis in-the-wild (ABAW) Competitioncited by 78
  5. 20247th ABAW Competition: Multi-Task Learning and Compound Expression Recognitioncited by 53
  6. 2022Transformer-based Multimodal Information Fusion for Facial Expression Analysiscited by 49
  7. 2022Vision Transformer with Attentive Pooling for Robust Facial Expression Recognitioncited by 35
  8. 2023Multi-modal Facial Affective Analysis based on Masked Autoencodercited by 23
  9. 2023Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognitioncited by 20
  10. 2022Coarse-to-Fine Cascaded Networks with Smooth Predicting for Video Facial Expression Recognitioncited by 17
  11. 2022Frame-level Prediction of Facial Expressions, Valence, Arousal and Action Units for Mobile Devicescited by 16
  12. 2022Multi-modal Multi-label Facial Action Unit Detection with Transformercited by 10
12 more
  1. 2023Spatio-Temporal AU Relational Graph Representation Learning For Facial Action Units Detectioncited by 7
  2. 2023Facial Affect Recognition based on Transformer Encoder and Audiovisual Fusion for the ABAW5 Challengecited by 6
  3. 2023A vector quantized masked autoencoder for audiovisual speech emotion recognitioncited by 6
  4. 2024Boosting Continuous Emotion Recognition with Self-Pretraining using Masked Autoencoders, Temporal Convolutional Networks, and Transformerscited by 6
  5. 2025Emotion Recognition with CLIP and Sequential Learningcited by 6
  6. 2022Facial Expression Classification using Fusion of Deep Neural Network in Video for the 3rd ABAW3 Competitioncited by 5
  7. 2022An Attention-based Method for Action Unit Detection at the 3rd ABAW Competitioncited by 4
  8. 2022Facial Action Unit Recognition Based on Transfer Learningcited by 4
  9. 2024Facial Affect Recognition based on Multi Architecture Encoder and Feature Fusion for the ABAW7 Challengecited by 2
  10. 2025Multi-task Adversarial Attacks against Black-box Model with Few-shot Queriescited by 1
  11. 2025ARPGNet: Appearance- and Relation-aware Parallel Graph Attention Fusion Network for Facial Expression Recognitioncited by 1
  12. 2021Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotationscited by 0