# Frame-level Prediction of Facial Expressions, Valence, Arousal and Action Units for Mobile Devices

arXiv: https://arxiv.org/abs/2203.13436 · HTML: https://mlsanity.com/paper/2203.13436/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace](https://mlsanity.com/paper/1910.04855/) (2019)
- [Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond](https://mlsanity.com/paper/1804.10938/) (2018)
- [Analysing Affective Behavior in the First ABAW 2020 Competition](https://mlsanity.com/paper/2001.11409/) (2020)
- [Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework](https://mlsanity.com/paper/2103.15792/) (2021)
- [Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study](https://mlsanity.com/paper/2105.03790/) (2021)
- [Analysing Affective Behavior in the second ABAW2 Competition](https://mlsanity.com/paper/2106.15318/) (2021)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges](https://mlsanity.com/paper/2202.10659/) (2022)
- [Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network](https://mlsanity.com/paper/1910.11111/) (2019)
- [Frame attention networks for facial expression recognition in videos](https://mlsanity.com/paper/1907.00193/) (2019)
- [Transformer-based Multimodal Information Fusion for Facial Expression Analysis](https://mlsanity.com/paper/2203.12367/) (2022)
- [Facial expression and attributes recognition based on multi-task learning of lightweight neural networks](https://mlsanity.com/paper/2103.17107/) (2021)
- [A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition](https://mlsanity.com/paper/2107.04187/) (2021)
- [Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition](https://mlsanity.com/paper/2108.04228/) (2021)
- [Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition](https://mlsanity.com/paper/2107.03708/) (2021)
- [Efficient Facial Representations for Age, Gender and Identity Recognition in Organizing Photo Albums using Multi-output CNN](https://mlsanity.com/paper/1807.07718/) (2018)
- [Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique](https://mlsanity.com/paper/2107.04192/) (2021)
- [Group-level Emotion Recognition using Transfer Learning from Face Identification](https://mlsanity.com/paper/1709.01688/) (2017)
- [Challenges of Artificial Intelligence -- From Machine Learning and Computer Vision to Emotional Intelligence](https://mlsanity.com/paper/2201.01466/) (2022)
- [Preferences Prediction using a Gallery of Mobile Device based on Scene Recognition and Object Detection](https://mlsanity.com/paper/1907.04519/) (2019)
- [Hotel Recognition via Latent Image Embedding](https://mlsanity.com/paper/2106.08042/) (2021)

## Built on by (most cited first, 16 of 16)

- [ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges](https://mlsanity.com/paper/2207.01138/) (2022)
- [ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges](https://mlsanity.com/paper/2303.01498/) (2023)
- [The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition](https://mlsanity.com/paper/2402.19344/) (2024)
- [7th ABAW Competition: Multi-Task Learning and Compound Expression Recognition](https://mlsanity.com/paper/2407.03835/) (2024)
- [A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition](https://mlsanity.com/paper/2203.14779/) (2022)
- [Facial Affective Behavior Analysis with Instruction Tuning](https://mlsanity.com/paper/2404.05052/) (2024)
- [Multi-modal Facial Affective Analysis based on Masked Autoencoder](https://mlsanity.com/paper/2303.10849/) (2023)
- [Joint Multimodal Transformer for Emotion Recognition in the Wild](https://mlsanity.com/paper/2403.10488/) (2024)
- [FaceCLIPNeRF: Text-driven 3D Face Manipulation using Deformable Neural Radiance Fields](https://mlsanity.com/paper/2307.11418/) (2023)
- [Affective Behavior Analysis using Action Unit Relation Graph and Multi-task Cross Attention](https://mlsanity.com/paper/2207.10293/) (2022)
- [A transformer-based approach to video frame-level prediction in Affective Behaviour Analysis In-the-wild](https://mlsanity.com/paper/2303.09293/) (2023)
- [Enhancing Facial Expression Recognition through Dual-Direction Attention Mixed Feature Networks: Application to 7th ABAW Challenge](https://mlsanity.com/paper/2407.12390/) (2024)
- [MAVEN: Multi-modal Attention for Valence-Arousal Emotion Network](https://mlsanity.com/paper/2503.12623/) (2025)
- [Computer Vision Estimation of Emotion Reaction Intensity in the Wild](https://mlsanity.com/paper/2303.10741/) (2023)
- [A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition](https://mlsanity.com/paper/2603.12221/) (2026)
- [Enhancing Facial Expression Recognition through Dual-Direction Attention Mixed Feature Networks and CLIP: Application to 8th ABAW Challenge](https://mlsanity.com/paper/2503.12260/) (2025)

## Abstract (arXiv)

In this paper, we consider the problem of real-time video-based facial emotion analytics, namely, facial expression recognition, prediction of valence and arousal and detection of action unit points. We propose the novel frame-level emotion recognition algorithm by extracting facial features with the single EfficientNet model pre-trained on AffectNet. As a result, our approach may be implemented even for video analytics on mobile devices. Experimental results for the large scale Aff-Wild2 database from the third Affective Behavior Analysis in-the-wild (ABAW) Competition demonstrate that our simple model is significantly better when compared to the VggFace baseline. In particular, our method is characterized by 0.15-0.2 higher performance measures for validation sets in uni-task Expression Classification, Valence-Arousal Estimation and Expression Classification. Due to simplicity, our approach may be considered as a new baseline for all four sub-challenges.