# Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study

arXiv: https://arxiv.org/abs/2105.03790 · HTML: https://mlsanity.com/paper/2105.03790/

## Key facts

- Cited by 168 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 64 references that are in the map.
- First posted on arXiv 8 May 2021.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [A Survey on Multi-Task Learning](https://mlsanity.com/paper/1707.08114/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Characterizing and Avoiding Negative Transfer](https://mlsanity.com/paper/1811.09751/) (2018)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [Understanding and Improving Information Transfer in Multi-Task Learning](https://mlsanity.com/paper/2005.00944/) (2020)
- [Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment](https://mlsanity.com/paper/1803.05588/) (2018)
- [Two-Stream Aural-Visual Affect Analysis in the Wild](https://mlsanity.com/paper/2002.03399/) (2020)
- [Facial Action Unit Detection Using Attention and Relation Learning](https://mlsanity.com/paper/1808.03457/) (2018)
- [An All-In-One Convolutional Neural Network for Face Analysis](https://mlsanity.com/paper/1611.00851/) (2016)
- [Heterogeneous Face Attribute Estimation: A Deep Multi-Task Learning Approach](https://mlsanity.com/paper/1706.00906/) (2017)
- [SSH: Single Stage Headless Face Detector](https://mlsanity.com/paper/1708.03979/) (2017)
- [Multitask Emotion Recognition with Incomplete Labels](https://mlsanity.com/paper/2002.03557/) (2020)
- [Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?](https://mlsanity.com/paper/1510.02969/) (2015)
- [FERA 2017 - Addressing Head Pose in the Third Facial Expression Recognition and Analysis Challenge](https://mlsanity.com/paper/1702.04174/) (2017)
- [EmotioNet Challenge: Recognition of facial expressions of emotion in the wild](https://mlsanity.com/paper/1703.01210/) (2017)
- [Two-level Attention with Two-stage Multi-task Learning for Facial Emotion Recognition](https://mlsanity.com/paper/1811.12139/) (2018)
- [Registration-free Face-SSD: Single shot analysis of smiles, facial attributes, and affect in the wild](https://mlsanity.com/paper/1902.04042/) (2019)
- [Attributes for Improved Attributes: A Multi-Task Network for Attribute Classification](https://mlsanity.com/paper/1604.07360/) (2016)

## Built on by (most cited first, 30 of 168)

- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [A Deep Neural Architecture for Harmonizing 3-D Input Data Analysis and Decision Making in Medical Imaging](https://mlsanity.com/paper/2303.00175/) (2023)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [MixAugment & Mixup: Augmentation Methods for Facial Expression Recognition](https://mlsanity.com/paper/2205.04442/) (2022)
- [Domain adaptation, Explainability & Fairness in AI for Medical Image Analysis: Diagnosis of COVID-19 based on 3-D Chest CT-scans](https://mlsanity.com/paper/2403.02192/) (2024)
- [Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos](https://mlsanity.com/paper/2112.00585/) (2021)
- [A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition](https://mlsanity.com/paper/2107.04187/) (2021)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [COVID-19 Computer-aided Diagnosis through AI-assisted CT Imaging Analysis: Deploying a Medical AI System](https://mlsanity.com/paper/2403.06242/) (2024)
- [Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition](https://mlsanity.com/paper/2108.04228/) (2021)
- [Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3](https://mlsanity.com/paper/2203.13031/) (2022)
- [AI-MIA: COVID-19 Detection & Severity Analysis through Medical Imaging](https://mlsanity.com/paper/2206.04732/) (2022)
- [Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition](https://mlsanity.com/paper/2303.08356/) (2023)
- [BTDNet: a Multi-Modal Approach for Brain Tumor Radiogenomic Classification](https://mlsanity.com/paper/2310.03485/) (2023)
- [A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis](https://mlsanity.com/paper/2107.04225/) (2021)
- [Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition](https://mlsanity.com/paper/2107.03708/) (2021)
- [LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis](https://mlsanity.com/paper/2308.10713/) (2023)
- [Affective Behaviour Analysis via Integrating Multi-Modal Knowledge](https://mlsanity.com/paper/2403.10825/) (2024)
- [Coarse-to-Fine Cascaded Networks with Smooth Predicting for Video Facial Expression Recognition](https://mlsanity.com/paper/2203.13052/) (2022)
- [Frame-level Prediction of Facial Expressions, Valence, Arousal and Action Units for Mobile Devices](https://mlsanity.com/paper/2203.13436/) (2022)
- [MMA-MRNNet: Harnessing Multiple Models of Affect and Dynamic Masked RNN for Precise Facial Expression Intensity Estimation](https://mlsanity.com/paper/2303.00180/) (2023)
- [Robust Light-Weight Facial Affective Behavior Recognition with CLIP](https://mlsanity.com/paper/2403.09915/) (2024)
- [Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit](https://mlsanity.com/paper/2409.17717/) (2024)

## Abstract (arXiv)

Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm, such as a DNN. MTL is based on the assumption that the tasks under consideration are related; therefore it exploits shared knowledge for improving performance on each individual task. Tasks are generally considered to be homogeneous, i.e., to refer to the same type of problem. Moreover, MTL is usually based on ground truth annotations with full, or partial overlap across tasks. In this work, we deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems. We explore task-relatedness as a means for co-training, in a weakly-supervised way, tasks that contain little, or even non-overlapping annotations. Task-relatedness is introduced in MTL, either explicitly through prior expert knowledge, or through data-driven studies. We propose a novel distribution matching approach, in which knowledge exchange is enabled between tasks, via matching of their predictions' distributions. Based on this approach, we build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks. We develop case studies for: i) continuous affect estimation, action unit detection, basic emotion recognition; ii) attribute detection, face identification. We illustrate that co-training via task relatedness alleviates negative transfer. Since FaceBehaviorNet learns features that encapsulate all aspects of facial behavior, we conduct zero-/few-shot learning to perform tasks beyond the ones that it has been trained for, such as compound emotion recognition. By conducting a very large experimental study, utilizing 10 databases, we illustrate that our approach outperforms, by large margins, the state-of-the-art in all tasks and in all databases, even in these which have not been used in its training.