# Controllable Image-to-Video Translation: A Case Study on Facial Expression Generation

arXiv: https://arxiv.org/abs/1808.02992 · HTML: https://mlsanity.com/paper/1808.02992/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Aug 2018.

## Built on by (most cited first, 12 of 12)

- [Targeted Supervised Contrastive Learning for Long-Tailed Recognition](https://mlsanity.com/paper/2111.13998/) (2021)
- [Sparse to Dense Dynamic 3D Facial Expression Generation](https://mlsanity.com/paper/2105.07463/) (2021)
- [Image-to-Video Generation via 3D Facial Dynamics](https://mlsanity.com/paper/2105.14678/) (2021)
- [Generating Multiple 4D Expression Transitions by Learning Face Landmark Trajectories](https://mlsanity.com/paper/2208.00050/) (2022)
- [Invertible Frowns: Video-to-Video Facial Emotion Translation](https://mlsanity.com/paper/2109.08061/) (2021)
- [AnimateMe: 4D Facial Expressions via Diffusion Models](https://mlsanity.com/paper/2403.17213/) (2024)
- [Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN](https://mlsanity.com/paper/2210.11182/) (2022)
- [Adaptively Enhancing Facial Expression Crucial Regions via Local Non-Local Joint Network](https://mlsanity.com/paper/2203.14045/) (2022)
- [Unpaired Multi-domain Attribute Translation of 3D Facial Shapes with a Square and Symmetric Geometric Map](https://mlsanity.com/paper/2308.13245/) (2023)
- [Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis](https://mlsanity.com/paper/2102.13329/) (2021)
- [GaFET: Learning Geometry-aware Facial Expression Translation from In-The-Wild Images](https://mlsanity.com/paper/2308.03413/) (2023)
- [Translation-based Video-to-Video Synthesis](https://mlsanity.com/paper/2404.04283/) (2024)

## Abstract (arXiv)

The recent advances in deep learning have made it possible to generate photo-realistic images by using neural networks and even to extrapolate video frames from an input video clip. In this paper, for the sake of both furthering this exploration and our own interest in a realistic application, we study image-to-video translation and particularly focus on the videos of facial expressions. This problem challenges the deep neural networks by another temporal dimension comparing to the image-to-image translation. Moreover, its single input image fails most existing video generation methods that rely on recurrent models. We propose a user-controllable approach so as to generate video clips of various lengths from a single face image. The lengths and types of the expressions are controlled by users. To this end, we design a novel neural network architecture that can incorporate the user input into its skip connections and propose several improvements to the adversarial training method for the neural network. Experiments and user studies verify the effectiveness of our approach. Especially, we would like to highlight that even for the face images in the wild (downloaded from the Web and the authors' own photos), our model can generate high-quality facial expression videos of which about 50% are labeled as real by Amazon Mechanical Turk workers.