# Image-to-Video Generation via 3D Facial Dynamics

arXiv: https://arxiv.org/abs/2105.14678 · HTML: https://mlsanity.com/paper/2105.14678/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 75 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [FSGAN: Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/1908.05932/) (2019)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [DRAW: A Recurrent Neural Network For Image Generation](https://mlsanity.com/paper/1502.04623/) (2015)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [GANimation: Anatomically-aware Facial Animation from a Single Image](https://mlsanity.com/paper/1807.09251/) (2018)
- [Semantic Image Inpainting with Deep Generative Models](https://mlsanity.com/paper/1607.07539/) (2016)
- [Video Summarization with Long Short-term Memory](https://mlsanity.com/paper/1605.08110/) (2016)
- [Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting](https://mlsanity.com/paper/1904.07475/) (2019)
- [Recycle-GAN: Unsupervised Video Retargeting](https://mlsanity.com/paper/1808.05174/) (2018)
- [Learning to Generate Long-term Future via Hierarchical Prediction](https://mlsanity.com/paper/1704.05831/) (2017)
- [The Pose Knows: Video Forecasting by Generating Pose Futures](https://mlsanity.com/paper/1705.00053/) (2017)
- [ReenactGAN: Learning to Reenact Faces via Boundary Transfer](https://mlsanity.com/paper/1807.11079/) (2018)
- [Neural Face Editing with Intrinsic Image Disentangling](https://mlsanity.com/paper/1704.04131/) (2017)
- [Deforming Autoencoders: Unsupervised Disentangling of Shape and Appearance](https://mlsanity.com/paper/1806.06503/) (2018)
- [Action-Agnostic Human Pose Forecasting](https://mlsanity.com/paper/1810.09676/) (2018)
- [Face Aging With Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1702.01983/) (2017)
- [CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training](https://mlsanity.com/paper/1703.10155/) (2017)
- [Geometry Guided Adversarial Facial Expression Synthesis](https://mlsanity.com/paper/1712.03474/) (2017)
- [Make a Face: Towards Arbitrary High Fidelity Face Manipulation](https://mlsanity.com/paper/1908.07191/) (2019)
- [Look Across Elapse: Disentangled Representation Learning and Photorealistic Cross-Age Face Synthesis for Age-Invariant Face Recognition](https://mlsanity.com/paper/1809.00338/) (2018)
- [Joint Face Image Restoration and Frontalization for Recognition](https://mlsanity.com/paper/2105.09907/) (2021)

## Built on by (most cited first, 13 of 13)

- [StableV2V: Stablizing Shape Consistency in Video-to-Video Editing](https://mlsanity.com/paper/2411.11045/) (2024)
- [More Real than Real: A Study on Human Visual Perception of Synthetic Faces](https://mlsanity.com/paper/2106.07226/) (2021)
- [Audio-Driven Dubbing for User Generated Contents via Style-Aware Semi-Parametric Synthesis](https://mlsanity.com/paper/2309.00030/) (2023)
- [Language-Guided Face Animation by Recurrent StyleGAN-based Generator](https://mlsanity.com/paper/2208.05617/) (2022)
- [3DPortraitGAN: Learning One-Quarter Headshot 3D GANs from a Single-View Portrait Dataset with Diverse Body Poses](https://mlsanity.com/paper/2307.14770/) (2023)
- [CorrTalk: Correlation Between Hierarchical Speech and Facial Activity Variances for 3D Animation](https://mlsanity.com/paper/2310.11295/) (2023)
- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [Facial Expression Video Generation Based-On Spatio-temporal Convolutional GAN: FEV-GAN](https://mlsanity.com/paper/2210.11182/) (2022)
- [Improving Transferable Targeted Adversarial Attack via Normalized Logit Calibration and Truncated Feature Mixing](https://mlsanity.com/paper/2405.06340/) (2024)
- [GenUDC: High Quality 3D Mesh Generation with Unsigned Dual Contouring Representation](https://mlsanity.com/paper/2410.17802/) (2024)
- [Towards Consistent and Controllable Image Synthesis for Face Editing](https://mlsanity.com/paper/2502.02465/) (2025)
- [Talking Surveys: How Photorealistic Embodied Conversational Agents Shape Response Quality, Engagement, and Satisfaction](https://mlsanity.com/paper/2508.02376/) (2025)
- [Speaking images. A novel framework for the automated self-description of artworks](https://mlsanity.com/paper/2506.05368/) (2025)

## Abstract (arXiv)

We present a versatile model, FaceAnime, for various video generation tasks from still images. Video generation from a single face image is an interesting problem and usually tackled by utilizing Generative Adversarial Networks (GANs) to integrate information from the input face image and a sequence of sparse facial landmarks. However, the generated face images usually suffer from quality loss, image distortion, identity change, and expression mismatching due to the weak representation capacity of the facial landmarks. In this paper, we propose to "imagine" a face video from a single face image according to the reconstructed 3D face dynamics, aiming to generate a realistic and identity-preserving face video, with precisely predicted pose and facial expression. The 3D dynamics reveal changes of the facial expression and motion, and can serve as a strong prior knowledge for guiding highly realistic face video generation. In particular, we explore face video prediction and exploit a well-designed 3D dynamic prediction network to predict a 3D dynamic sequence for a single face image. The 3D dynamics are then further rendered by the sparse texture mapping algorithm to recover structural details and sparse textures for generating face frames. Our model is versatile for various AR/VR and entertainment applications, such as face video retargeting and face video prediction. Superior experimental results have well demonstrated its effectiveness in generating high-fidelity, identity-preserving, and visually pleasant face video clips from a single source face image.