# Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN

arXiv: https://arxiv.org/abs/2210.11182 · HTML: https://mlsanity.com/paper/2210.11182/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 60 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [GANimation: Anatomically-aware Facial Animation from a Single Image](https://mlsanity.com/paper/1807.09251/) (2018)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [Talking-head Generation with Rhythmic Head Motion](https://mlsanity.com/paper/2007.08547/) (2020)
- [MarioNETte: Few-shot Face Reenactment Preserving Identity of Unseen Targets](https://mlsanity.com/paper/1911.08139/) (2019)
- [One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning](https://mlsanity.com/paper/2112.02749/) (2021)
- [Imitating Arbitrary Talking Style for Realistic Audio-DrivenTalking Face Synthesis](https://mlsanity.com/paper/2111.00203/) (2021)
- [ExprGAN: Facial Expression Editing with Controllable Expression Intensity](https://mlsanity.com/paper/1709.03842/) (2017)
- [Cascade EF-GAN: Progressive Facial Expression Editing with Local Focuses](https://mlsanity.com/paper/2003.05905/) (2020)
- [Emotion-Controllable Generalized Talking Face Generation](https://mlsanity.com/paper/2205.01155/) (2022)
- [Geometry Guided Adversarial Facial Expression Synthesis](https://mlsanity.com/paper/1712.03474/) (2017)
- [Learning to Forecast and Refine Residual Motion for Image-to-Video Generation](https://mlsanity.com/paper/1807.09951/) (2018)
- [Every Smile is Unique: Landmark-Guided Diverse Smile Generation](https://mlsanity.com/paper/1802.01873/) (2018)
- [Dynamic Facial Expression Generation on Hilbert Hypersphere with Conditional Wasserstein Generative Adversarial Nets](https://mlsanity.com/paper/1907.10087/) (2019)
- [Video Prediction with Appearance and Motion Conditions](https://mlsanity.com/paper/1807.02635/) (2018)
- [Image-to-Video Generation via 3D Facial Dynamics](https://mlsanity.com/paper/2105.14678/) (2021)
- [Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation](https://mlsanity.com/paper/1808.02992/) (2018)

## Built on by (most cited first, 4 of 4)

- [4D Facial Expression Diffusion Model](https://mlsanity.com/paper/2303.16611/) (2023)
- [AnimateMe: 4D Facial Expressions via Diffusion Models](https://mlsanity.com/paper/2403.17213/) (2024)
- [FacEnhance: Facial Expression Enhancing with Recurrent DDPMs](https://mlsanity.com/paper/2406.09040/) (2024)
- [How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations](https://mlsanity.com/paper/2409.02566/) (2024)

## Abstract (arXiv)

Facial expression generation has always been an intriguing task for scientists and researchers all over the globe. In this context, we present our novel approach for generating videos of the six basic facial expressions. Starting from a single neutral facial image and a label indicating the desired facial expression, we aim to synthesize a video of the given identity performing the specified facial expression. Our approach, referred to as FEV-GAN (Facial Expression Video GAN), is based on Spatio-temporal Convolutional GANs, that are known to model both content and motion in the same network. Previous methods based on such a network have shown a good ability to generate coherent videos with smooth temporal evolution. However, they still suffer from low image quality and low identity preservation capability. In this work, we address this problem by using a generator composed of two image encoders. The first one is pre-trained for facial identity feature extraction and the second for spatial feature extraction. We have qualitatively and quantitatively evaluated our model on two international facial expression benchmark databases: MUG and Oulu-CASIA NIR&VIS. The experimental results analysis demonstrates the effectiveness of our approach in generating videos of the six basic facial expressions while preserving the input identity. The analysis also proves that the use of both identity and spatial features enhances the decoder ability to better preserve the identity and generate high-quality videos. The code and the pre-trained model will soon be made publicly available.