# Iterative Text-based Editing of Talking-heads Using Neural Retargeting

arXiv: https://arxiv.org/abs/2011.10688 · HTML: https://mlsanity.com/paper/2011.10688/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2020.

## Built on by (most cited first, 21 of 21)

- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation](https://mlsanity.com/paper/2109.10595/) (2021)
- [FSGANv2: Improved Subject Agnostic Face Swapping and Reenactment](https://mlsanity.com/paper/2202.12972/) (2022)
- [Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos](https://mlsanity.com/paper/2112.00585/) (2021)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [PersonaTalk: Bring Attention to Your Persona in Visual Dubbing](https://mlsanity.com/paper/2409.05379/) (2024)
- [ExpressEdit: Video Editing with Natural Language and Sketching](https://mlsanity.com/paper/2403.17693/) (2024)
- [StableFace: Analyzing and Improving Motion Stability for Talking Face Generation](https://mlsanity.com/paper/2208.13717/) (2022)
- [Invertible Frowns: Video-to-Video Facial Emotion Translation](https://mlsanity.com/paper/2109.08061/) (2021)
- [READ Avatars: Realistic Emotion-controllable Audio Driven Avatars](https://mlsanity.com/paper/2303.00744/) (2023)
- [MusicFace: Music-driven Expressive Singing Face Synthesis](https://mlsanity.com/paper/2303.14044/) (2023)
- [Express4D: Expressive, Friendly, and Extensible 4D Facial Motion Generation Benchmark](https://mlsanity.com/paper/2508.12438/) (2025)
- [Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism](https://mlsanity.com/paper/2312.06613/) (2023)
- [Text-based Talking Video Editing with Cascaded Conditional Diffusion](https://mlsanity.com/paper/2407.14841/) (2024)
- [Towards MOOCs for Lipreading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale](https://mlsanity.com/paper/2208.09796/) (2022)
- [EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers](https://mlsanity.com/paper/2601.22127/) (2026)
- [Neural Relighting and Expression Transfer On Video Portraits](https://mlsanity.com/paper/2107.14735/) (2021)
- [Intelligent Video Editing: Incorporating Modern Talking Face Generation Algorithms in a Video Editor](https://mlsanity.com/paper/2110.08580/) (2021)
- [Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors](https://mlsanity.com/paper/2208.08118/) (2022)
- [All's well that FID's well? Result quality and metric scores in GAN models for lip-sychronization tasks](https://mlsanity.com/paper/2212.13810/) (2022)
- [FacEDiT: Unified Talking Face Editing and Generation via Facial Motion Infilling](https://mlsanity.com/paper/2512.14056/) (2025)

## Abstract (arXiv)

We present a text-based tool for editing talking-head video that enables an iterative editing workflow. On each iteration users can edit the wording of the speech, further refine mouth motions if necessary to reduce artifacts and manipulate non-verbal aspects of the performance by inserting mouth gestures (e.g. a smile) or changing the overall performance style (e.g. energetic, mumble). Our tool requires only 2-3 minutes of the target actor video and it synthesizes the video for each iteration in about 40 seconds, allowing users to quickly explore many editing possibilities as they iterate. Our approach is based on two key ideas. (1) We develop a fast phoneme search algorithm that can quickly identify phoneme-level subsequences of the source repository video that best match a desired edit. This enables our fast iteration loop. (2) We leverage a large repository of video of a source actor and develop a new self-supervised neural retargeting technique for transferring the mouth motions of the source actor to the target actor. This allows us to work with relatively short target actor videos, making our approach applicable in many real-world editing scenarios. Finally, our refinement and performance controls give users the ability to further fine-tune the synthesized results.