# Realistic Face Reenactment via Self-Supervised Disentangling of Identity and Pose

arXiv: https://arxiv.org/abs/2003.12957 · HTML: https://mlsanity.com/paper/2003.12957/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2020.

## Built on by (most cited first, 26 of 26)

- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis](https://mlsanity.com/paper/2211.14506/) (2022)
- [One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning](https://mlsanity.com/paper/2112.02749/) (2021)
- [Deepfake Generation and Detection: A Benchmark and Survey](https://mlsanity.com/paper/2403.17881/) (2024)
- [Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Control](https://mlsanity.com/paper/2405.12970/) (2024)
- [Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors](https://mlsanity.com/paper/2212.04248/) (2022)
- [Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation](https://mlsanity.com/paper/2104.07995/) (2021)
- [Region-Aware Face Swapping](https://mlsanity.com/paper/2203.04564/) (2022)
- [High-Fidelity and Freely Controllable Talking Head Video Generation](https://mlsanity.com/paper/2304.10168/) (2023)
- [Test-Time Personalization with a Transformer for Human Pose Estimation](https://mlsanity.com/paper/2107.02133/) (2021)
- [Finding Directions in GAN's Latent Space for Neural Face Reenactment](https://mlsanity.com/paper/2202.00046/) (2022)
- [FaceController: Controllable Attribute Editing for Face in the Wild](https://mlsanity.com/paper/2102.11464/) (2021)
- [Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis](https://mlsanity.com/paper/2109.02081/) (2021)
- [A Generalist FaceX via Learning Unified Facial Representation](https://mlsanity.com/paper/2401.00551/) (2023)
- [Multimodal-driven Talking Face Generation via a Unified Diffusion-based Generator](https://mlsanity.com/paper/2305.02594/) (2023)
- [A Review of Human Emotion Synthesis Based on Generative Technology](https://mlsanity.com/paper/2412.07116/) (2024)
- [One-shot Neural Face Reenactment via Finding Directions in GAN's Latent Space](https://mlsanity.com/paper/2402.03553/) (2024)
- [OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions](https://mlsanity.com/paper/2309.16148/) (2023)
- [Expressive Talking Head Video Encoding in StyleGAN2 Latent-Space](https://mlsanity.com/paper/2203.14512/) (2022)
- [A comprehensive survey on semantic facial attribute editing using generative adversarial networks](https://mlsanity.com/paper/2205.10587/) (2022)
- [One-Shot Face Video Re-enactment using Hybrid Latent Spaces of StyleGAN2](https://mlsanity.com/paper/2302.07848/) (2023)
- [SelFSR: Self-Conditioned Face Super-Resolution in the Wild via Flow Field Degradation Network](https://mlsanity.com/paper/2112.10683/) (2021)
- [PortraitDirector: A Hierarchical Disentanglement Framework for Controllable and Real-time Facial Reenactment](https://mlsanity.com/paper/2604.19129/) (2026)
- [Video-Driven Animation of Neural Head Avatars](https://mlsanity.com/paper/2403.04380/) (2024)
- [PerformRecast: Expression and Head Pose Disentanglement for Portrait Video Editing](https://mlsanity.com/paper/2603.19731/) (2026)

## Abstract (arXiv)

Recent works have shown how realistic talking face images can be obtained under the supervision of geometry guidance, e.g., facial landmark or boundary. To alleviate the demand for manual annotations, in this paper, we propose a novel self-supervised hybrid model (DAE-GAN) that learns how to reenact face naturally given large amounts of unlabeled videos. Our approach combines two deforming autoencoders with the latest advances in the conditional generation. On the one hand, we adopt the deforming autoencoder to disentangle identity and pose representations. A strong prior in talking face videos is that each frame can be encoded as two parts: one for video-specific identity and the other for various poses. Inspired by that, we utilize a multi-frame deforming autoencoder to learn a pose-invariant embedded face for each video. Meanwhile, a multi-scale deforming autoencoder is proposed to extract pose-related information for each frame. On the other hand, the conditional generator allows for enhancing fine details and overall reality. It leverages the disentangled features to generate photo-realistic and pose-alike face images. We evaluate our model on VoxCeleb1 and RaFD dataset. Experiment results demonstrate the superior quality of reenacted images and the flexibility of transferring facial movements between identities.