# Deep Semantic Manipulation of Facial Videos

arXiv: https://arxiv.org/abs/2111.07902 · HTML: https://mlsanity.com/paper/2111.07902/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [FaceForensics++: Learning to Detect Manipulated Facial Images](https://mlsanity.com/paper/1901.08971/) (2019)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [StyleRig: Rigging StyleGAN for 3D Control over Portrait Images](https://mlsanity.com/paper/2004.00121/) (2020)
- [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Deep Photo Style Transfer](https://mlsanity.com/paper/1703.07511/) (2017)
- [ICface: Interpretable and Controllable Face Reenactment Using GANs](https://mlsanity.com/paper/1904.01909/) (2019)
- [Neural Style-Preserving Visual Dubbing](https://mlsanity.com/paper/1909.02518/) (2019)
- [Automatic Face Reenactment](https://mlsanity.com/paper/1602.02651/) (2016)
- [Head2Head: Video-based Neural Head Synthesis](https://mlsanity.com/paper/2005.10954/) (2020)
- [Deepfake Detection using Spatiotemporal Convolutional Networks](https://mlsanity.com/paper/2006.14749/) (2020)
- [Head2Head++: Deep Facial Attributes Re-Targeting](https://mlsanity.com/paper/2006.10199/) (2020)
- [Real-time Facial Expression Recognition "In The Wild'' by Disentangling 3D Expression from Identity](https://mlsanity.com/paper/2005.05509/) (2020)
- [Deep Covariance Descriptors for Facial Expression Recognition](https://mlsanity.com/paper/1805.03869/) (2018)
- [Head2HeadFS: Video-based Head Reenactment with Few-shot Learning](https://mlsanity.com/paper/2103.16229/) (2021)

## Built on by (most cited first, 6 of 6)

- [Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos](https://mlsanity.com/paper/2112.00585/) (2021)
- [Continuously Controllable Facial Expression Editing in Talking Face Videos](https://mlsanity.com/paper/2209.08289/) (2022)
- [Contrastive Decoupled Representation Learning and Regularization for Speech-Preserving Facial Expression Manipulation](https://mlsanity.com/paper/2504.05672/) (2025)
- [Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism](https://mlsanity.com/paper/2312.06613/) (2023)
- [Learning Spatial-Temporal Coherent Correlations for Speech-Preserving Facial Expression Manipulation](https://mlsanity.com/paper/2604.20226/) (2026)
- [Exploring Talking Head Models With Adjacent Frame Prior for Speech-Preserving Facial Expression Manipulation](https://mlsanity.com/paper/2601.12876/) (2026)

## Abstract (arXiv)

Editing and manipulating facial features in videos is an interesting and important field of research with a plethora of applications, ranging from movie post-production and visual effects to realistic avatars for video games and virtual assistants. Our method supports semantic video manipulation based on neural rendering and 3D-based facial expression modelling. We focus on interactive manipulation of the videos by altering and controlling the facial expressions, achieving promising photorealistic results. The proposed method is based on a disentangled representation and estimation of the 3D facial shape and activity, providing the user with intuitive and easy-to-use control of the facial expressions in the input video. We also introduce a user-friendly, interactive AI tool that processes human-readable semantic labels about the desired expression manipulations in specific parts of the input video and synthesizes photorealistic manipulated videos. We achieve that by mapping the emotion labels to points on the Valence-Arousal space (where Valence quantifies how positive or negative is an emotion and Arousal quantifies the power of the emotion activation), which in turn are mapped to disentangled 3D facial expressions through an especially-designed and trained expression decoder network. The paper presents detailed qualitative and quantitative experiments, which demonstrate the effectiveness of our system and the promising results it achieves.