# Identity-Preserving Realistic Talking Face Generation

arXiv: https://arxiv.org/abs/2005.12318 · HTML: https://mlsanity.com/paper/2005.12318/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2020.

## Built on by (most cited first, 12 of 12)

- [FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning](https://mlsanity.com/paper/2108.07938/) (2021)
- [Emotion-Controllable Generalized Talking Face Generation](https://mlsanity.com/paper/2205.01155/) (2022)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [Speech2Video: Cross-Modal Distillation for Speech to Video Generation](https://mlsanity.com/paper/2107.04806/) (2021)
- [Emotional Talking Head Generation based on Memory-Sharing and Attention-Augmented Networks](https://mlsanity.com/paper/2306.03594/) (2023)
- [MusicFace: Music-driven Expressive Singing Face Synthesis](https://mlsanity.com/paper/2303.14044/) (2023)
- [GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptation](https://mlsanity.com/paper/2506.21513/) (2025)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Laughing Matters: Introducing Laughing-Face Generation using Diffusion Models](https://mlsanity.com/paper/2305.08854/) (2023)
- [DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model](https://mlsanity.com/paper/2503.19001/) (2025)
- [Adversarial Identity Injection for Semantic Face Image Synthesis](https://mlsanity.com/paper/2404.10408/) (2024)
- [Controllable Expressive 3D Facial Animation via Diffusion in a Unified Multimodal Space](https://mlsanity.com/paper/2506.10007/) (2025)

## Abstract (arXiv)

Speech-driven facial animation is useful for a variety of applications such as telepresence, chatbots, etc. The necessary attributes of having a realistic face animation are 1) audio-visual synchronization (2) identity preservation of the target individual (3) plausible mouth movements (4) presence of natural eye blinks. The existing methods mostly address the audio-visual lip synchronization, and few recent works have addressed the synthesis of natural eye blinks for overall video realism. In this paper, we propose a method for identity-preserving realistic facial animation from speech. We first generate person-independent facial landmarks from audio using DeepSpeech features for invariance to different voices, accents, etc. To add realism, we impose eye blinks on facial landmarks using unsupervised learning and retargets the person-independent landmarks to person-specific landmarks to preserve the identity-related facial structure which helps in the generation of plausible mouth shapes of the target identity. Finally, we use LSGAN to generate the facial texture from person-specific facial landmarks, using an attention mechanism that helps to preserve identity-related texture. An extensive comparison of our proposed method with the current state-of-the-art methods demonstrates a significant improvement in terms of lip synchronization accuracy, image reconstruction quality, sharpness, and identity-preservation. A user study also reveals improved realism of our animation results over the state-of-the-art methods. To the best of our knowledge, this is the first work in speech-driven 2D facial animation that simultaneously addresses all the above-mentioned attributes of a realistic speech-driven face animation.