# Multi-Task Learning Framework for Emotion Recognition in-the-wild

arXiv: https://arxiv.org/abs/2207.09373 · HTML: https://mlsanity.com/paper/2207.09373/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Partial FC: Training 10 Million Identities on a Single Machine](https://mlsanity.com/paper/2010.05222/) (2020)
- [Training Deep Networks for Facial Expression Recognition with Crowd-Sourced Label Distribution](https://mlsanity.com/paper/1608.01041/) (2016)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1402.1128/) (2014)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition](https://mlsanity.com/paper/1811.07770/) (2018)
- [Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition](https://mlsanity.com/paper/2109.07270/) (2021)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [Photorealistic Facial Synthesis in the Dimensional Affect Space](https://mlsanity.com/paper/1811.08004/) (2018)
- [Facial Action Unit Intensity Estimation via Semantic Correspondence Learning with Dynamic Graph Convolution](https://mlsanity.com/paper/2004.09681/) (2020)
- [Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3](https://mlsanity.com/paper/2203.13031/) (2022)
- [An Ensemble Approach for Facial Expression Analysis in Video](https://mlsanity.com/paper/2203.12891/) (2022)

## Built on by (most cited first, 10 of 10)

- [Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning](https://mlsanity.com/paper/2504.07198/) (2025)
- [A transformer-based approach to video frame-level prediction in Affective Behaviour Analysis In-the-wild](https://mlsanity.com/paper/2303.09293/) (2023)
- [Affective Behaviour Analysis via Progressive Learning](https://mlsanity.com/paper/2407.16945/) (2024)
- [HSEmotion Team at the 7th ABAW Challenge: Multi-Task Learning and Compound Facial Expression Recognition](https://mlsanity.com/paper/2407.13184/) (2024)
- [Exploring Expression-related Self-supervised Learning for Affective Behaviour Analysis](https://mlsanity.com/paper/2303.10511/) (2023)
- [Efficient Labelling of Affective Video Datasets via Few-Shot & Multi-Task Contrastive Learning](https://mlsanity.com/paper/2308.02173/) (2023)
- [MVP: Multimodal Emotion Recognition based on Video and Physiological Signals](https://mlsanity.com/paper/2501.03103/) (2025)
- [MTCAE-DFER: Multi-Task Cascaded Autoencoder for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2412.18988/) (2024)
- [HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition](https://mlsanity.com/paper/2607.12774/) (2026)
- [7ABAW-Compound Expression Recognition via Curriculum Learning](https://mlsanity.com/paper/2503.07969/) (2025)

## Abstract (arXiv)

This paper presents our system for the Multi-Task Learning (MTL) Challenge in the 4th Affective Behavior Analysis in-the-wild (ABAW) competition. We explore the research problems of this challenge from three aspects: 1) For obtaining efficient and robust visual feature representations, we propose MAE-based unsupervised representation learning and IResNet/DenseNet-based supervised representation learning methods; 2) Considering the importance of temporal information in videos, we explore three types of sequential encoders to capture the temporal information, including the encoder based on transformer, the encoder based on LSTM, and the encoder based on GRU; 3) For modeling the correlation between these different tasks (i.e., valence, arousal, expression, and AU) for multi-task affective analysis, we first explore the dependency between these different tasks and propose three multi-task learning frameworks to model the correlations effectively. Our system achieves the performance of 1.7607 on the validation dataset and 1.4361 on the test dataset, ranking first in the MTL Challenge. The code is available at https://github.com/AIM3-RUC/ABAW4.