# Facial expression and attributes recognition based on multi-task learning of lightweight neural networks

arXiv: https://arxiv.org/abs/2103.17107 · HTML: https://mlsanity.com/paper/2103.17107/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [High-Performance Large-Scale Image Recognition Without Normalization](https://mlsanity.com/paper/2102.06171/) (2021)
- [Compacting, Picking and Growing for Unforgetting Continual Learning](https://mlsanity.com/paper/1910.06562/) (2019)
- [Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition](https://mlsanity.com/paper/1905.04075/) (2019)
- [Frame attention networks for facial expression recognition in videos](https://mlsanity.com/paper/1907.00193/) (2019)
- [Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2103.09154/) (2021)
- [Efficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural Networks](https://mlsanity.com/paper/2001.06338/) (2020)
- [Learning to Amend Facial Expression Representation via De-albino and Affinity](https://mlsanity.com/paper/2103.10189/) (2021)
- [Temporal Multimodal Fusion for Video Emotion Classification in the Wild](https://mlsanity.com/paper/1709.07200/) (2017)
- [Efficient Facial Representations for Age, Gender and Identity Recognition in Organizing Photo Albums using Multi-output CNN](https://mlsanity.com/paper/1807.07718/) (2018)
- [Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video](https://mlsanity.com/paper/1711.04598/) (2017)
- [Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition](https://mlsanity.com/paper/2008.02655/) (2020)
- [Emotion Recognition with Spatial Attention and Temporal Softmax Pooling](https://mlsanity.com/paper/1910.01254/) (2019)
- [Computational Emotion Analysis From Images: Recent Advances and Future Directions](https://mlsanity.com/paper/2103.10798/) (2021)
- [CAKE: Compact and Accurate K-dimensional representation of Emotion](https://mlsanity.com/paper/1807.11215/) (2018)

## Built on by (most cited first, 30 of 40)

- [Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond](https://mlsanity.com/paper/2401.01219/) (2024)
- [Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition](https://mlsanity.com/paper/2109.07270/) (2021)
- [POSTER++: A simpler and stronger facial expression recognition network](https://mlsanity.com/paper/2301.12149/) (2023)
- [Frame-level Prediction of Facial Expressions, Valence, Arousal and Action Units for Mobile Devices](https://mlsanity.com/paper/2203.13436/) (2022)
- [Continuously Controllable Facial Expression Editing in Talking Face Videos](https://mlsanity.com/paper/2209.08289/) (2022)
- [Latent-OFER: Detect, Mask, and Reconstruct with Latent Vectors for Occluded Facial Expression Recognition](https://mlsanity.com/paper/2307.11404/) (2023)
- [VEATIC: Video-based Emotion and Affect Tracking in Context Dataset](https://mlsanity.com/paper/2309.06745/) (2023)
- [Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation](https://mlsanity.com/paper/2403.02302/) (2024)
- [Affective Behavior Analysis using Action Unit Relation Graph and Multi-task Cross Attention](https://mlsanity.com/paper/2207.10293/) (2022)
- [Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss](https://mlsanity.com/paper/2207.11482/) (2022)
- [GOTCHA: Real-Time Video Deepfake Detection via Challenge-Response](https://mlsanity.com/paper/2210.06186/) (2022)
- [EmotiEffNet Facial Features in Uni-task Emotion Recognition in Video at ABAW-5 competition](https://mlsanity.com/paper/2303.09162/) (2023)
- [HSE-NN Team at the 4th ABAW Competition: Multi-task Emotion Recognition and Learning from Synthetic Images](https://mlsanity.com/paper/2207.09508/) (2022)
- [Cluster-level pseudo-labelling for source-free cross-domain facial expression recognition](https://mlsanity.com/paper/2210.05246/) (2022)
- [Photorealistic and Identity-Preserving Image-Based Emotion Manipulation with Latent Diffusion Models](https://mlsanity.com/paper/2308.03183/) (2023)
- [Neuromorphic Valence and Arousal Estimation](https://mlsanity.com/paper/2401.16058/) (2024)
- [Headset: Human emotion awareness under partial occlusions multimodal dataset](https://mlsanity.com/paper/2402.09107/) (2024)
- [Multimodal End-to-End Group Emotion Recognition using Cross-Modal Attention](https://mlsanity.com/paper/2111.05890/) (2021)
- [A Study on the Ambiguity in Human Annotation of German Oral History Interviews for Perceived Emotion Recognition and Sentiment Analysis](https://mlsanity.com/paper/2201.06868/) (2022)
- [Detecting Interlocutor Confusion in Situated Human-Avatar Dialogue: A Pilot Study](https://mlsanity.com/paper/2206.02436/) (2022)
- [EmojiHeroVR: A Study on Facial Expression Recognition under Partial Occlusion from Head-Mounted Displays](https://mlsanity.com/paper/2410.03331/) (2024)
- [A Comprehensive Study of Deep Learning Model Fixing Approaches](https://mlsanity.com/paper/2512.23745/) (2025)
- [MobileAgeNet: Lightweight Facial Age Estimation for Mobile Deployment](https://mlsanity.com/paper/2604.17007/) (2026)
- [Comparing Facial Expression Recognition in Humans and Machines: Using CAM, GradCAM, and Extremal Perturbation](https://mlsanity.com/paper/2110.04481/) (2021)
- [Affect-DML: Context-Aware One-Shot Recognition of Human Affect using Deep Metric Learning](https://mlsanity.com/paper/2111.15271/) (2021)
- [Transferring Studies Across Embodiments: A Case Study in Confusion Detection](https://mlsanity.com/paper/2206.01493/) (2022)
- [A Survey of Face Recognition](https://mlsanity.com/paper/2212.13038/) (2022)
- [Detail-Enhanced Intra- and Inter-modal Interaction for Audio-Visual Emotion Recognition](https://mlsanity.com/paper/2405.16701/) (2024)
- [Nonverbal Immediacy Analysis in Education: A Multimodal Computational Model](https://mlsanity.com/paper/2407.17209/) (2024)
- [ID-Consistent, Precise Expression Generation with Blendshape-Guided Diffusion](https://mlsanity.com/paper/2510.04706/) (2025)

## Abstract (arXiv)

In this paper, the multi-task learning of lightweight convolutional neural networks is studied for face identification and classification of facial attributes (age, gender, ethnicity) trained on cropped faces without margins. The necessity to fine-tune these networks to predict facial expressions is highlighted. Several models are presented based on MobileNet, EfficientNet and RexNet architectures. It was experimentally demonstrated that they lead to near state-of-the-art results in age, gender and race recognition on the UTKFace dataset and emotion classification on the AffectNet dataset. Moreover, it is shown that the usage of the trained models as feature extractors of facial regions in video frames leads to 4.5% higher accuracy than the previously known state-of-the-art single models for the AFEW and the VGAF datasets from the EmotiW challenges. The models and source code are publicly available at https://github.com/HSE-asavchenko/face-emotion-recognition.