# eMotion-GAN: A Motion-based GAN for Photorealistic and Facial Expression Preserving Frontal View Synthesis

arXiv: https://arxiv.org/abs/2404.09940 · HTML: https://mlsanity.com/paper/2404.09940/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 72 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Deep Facial Expression Recognition: A Survey](https://mlsanity.com/paper/1804.08348/) (2018)
- [Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis](https://mlsanity.com/paper/1701.02096/) (2017)
- [Video Frame Interpolation via Adaptive Separable Convolution](https://mlsanity.com/paper/1708.01692/) (2017)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition](https://mlsanity.com/paper/2104.00232/) (2021)
- [Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis](https://mlsanity.com/paper/1704.04086/) (2017)
- [Distract Your Attention: Multi-head Cross Attention Network for Facial Expression Recognition](https://mlsanity.com/paper/2109.07270/) (2021)
- [Towards Large-Pose Face Frontalization in the Wild](https://mlsanity.com/paper/1704.06244/) (2017)
- [Cascade EF-GAN: Progressive Facial Expression Editing with Local Focuses](https://mlsanity.com/paper/2003.05905/) (2020)
- [DRB-GAN: A Dynamic ResBlock Generative Adversarial Network for Artistic Style Transfer](https://mlsanity.com/paper/2108.07379/) (2021)
- [Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images](https://mlsanity.com/paper/2003.08124/) (2020)
- [Supervision by Registration and Triangulation for Landmark Detection](https://mlsanity.com/paper/2101.09866/) (2021)
- [Learning Facial Representations from the Cycle-consistency of Face](https://mlsanity.com/paper/2108.03427/) (2021)
- [Impact of facial landmark localization on facial expression recognition](https://mlsanity.com/paper/1905.10784/) (2019)
- [Dynamic Facial Expression Recognition under Partial Occlusion with Optical Flow Reconstruction](https://mlsanity.com/paper/2012.13217/) (2020)
- [Optical Flow Techniques for Facial Expression Analysis -- a Practical Evaluation Study](https://mlsanity.com/paper/1904.11592/) (2019)
- [DeepFaceFlow: In-the-wild Dense 3D Facial Motion Estimation](https://mlsanity.com/paper/2005.07298/) (2020)
- [Expression-preserving face frontalization improves visually assisted speech processing](https://mlsanity.com/paper/2204.02810/) (2022)

## Abstract (arXiv)

Many existing facial expression recognition (FER) systems encounter substantial performance degradation when faced with variations in head pose. Numerous frontalization methods have been proposed to enhance these systems' performance under such conditions. However, they often introduce undesirable deformations, rendering them less suitable for precise facial expression analysis. In this paper, we present eMotion-GAN, a novel deep learning approach designed for frontal view synthesis while preserving facial expressions within the motion domain. Considering the motion induced by head variation as noise and the motion induced by facial expression as the relevant information, our model is trained to filter out the noisy motion in order to retain only the motion related to facial expression. The filtered motion is then mapped onto a neutral frontal face to generate the corresponding expressive frontal face. We conducted extensive evaluations using several widely recognized dynamic FER datasets, which encompass sequences exhibiting various degrees of head pose variations in both intensity and orientation. Our results demonstrate the effectiveness of our approach in significantly reducing the FER performance gap between frontal and non-frontal faces. Specifically, we achieved a FER improvement of up to +5% for small pose variations and up to +20% improvement for larger pose variations. Code available at urlhttps://github.com/o-ikne/eMotion-GAN.git.