# Joint Deep Learning of Facial Expression Synthesis and Recognition

arXiv: https://arxiv.org/abs/2002.02194 · HTML: https://mlsanity.com/paper/2002.02194/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Feb 2020.

## Built on by (most cited first, 6 of 6)

- [4D Facial Expression Diffusion Model](https://mlsanity.com/paper/2303.16611/) (2023)
- [Towards Unbiased Visual Emotion Recognition via Causal Intervention](https://mlsanity.com/paper/2107.12096/) (2021)
- [MagicFace: High-Fidelity Facial Expression Editing with Action-Unit Control](https://mlsanity.com/paper/2501.02260/) (2025)
- [FacEnhance: Facial Expression Enhancing with Recurrent DDPMs](https://mlsanity.com/paper/2406.09040/) (2024)
- [How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations](https://mlsanity.com/paper/2409.02566/) (2024)
- [Synthetic Data Generation for Emotional Depth Faces: Optimizing Conditional DCGANs via Genetic Algorithms in the Latent Space and Stabilizing Training with Knowledge Distillation](https://mlsanity.com/paper/2508.09188/) (2025)

## Abstract (arXiv)

Recently, deep learning based facial expression recognition (FER) methods have attracted considerable attention and they usually require large-scale labelled training data. Nonetheless, the publicly available facial expression databases typically contain a small amount of labelled data. In this paper, to overcome the above issue, we propose a novel joint deep learning of facial expression synthesis and recognition method for effective FER. More specifically, the proposed method involves a two-stage learning procedure. Firstly, a facial expression synthesis generative adversarial network (FESGAN) is pre-trained to generate facial images with different facial expressions. To increase the diversity of the training images, FESGAN is elaborately designed to generate images with new identities from a prior distribution. Secondly, an expression recognition network is jointly learned with the pre-trained FESGAN in a unified framework. In particular, the classification loss computed from the recognition network is used to simultaneously optimize the performance of both the recognition network and the generator of FESGAN. Moreover, in order to alleviate the problem of data bias between the real images and the synthetic images, we propose an intra-class loss with a novel real data-guided back-propagation (RDBP) algorithm to reduce the intra-class variations of images from the same class, which can significantly improve the final performance. Extensive experimental results on public facial expression databases demonstrate the superiority of the proposed method compared with several state-of-the-art FER methods.