# Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation

arXiv: https://arxiv.org/abs/2507.20953 · HTML: https://mlsanity.com/paper/2507.20953/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 83 of its 113 references that are in the map.
- First posted on arXiv 28 Jul 2025.

## Builds on (most cited first, 30 of 83)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [Talking Face Generation by Adversarially Disentangled Audio-Visual Representation](https://mlsanity.com/paper/1807.07860/) (2018)
- [Self-Supervised Learning of Audio-Visual Objects from Video](https://mlsanity.com/paper/2008.04237/) (2020)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN](https://mlsanity.com/paper/2203.04036/) (2022)
- [EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model](https://mlsanity.com/paper/2205.15278/) (2022)
- [Realistic Speech-Driven Facial Animation with GANs](https://mlsanity.com/paper/1906.06337/) (2019)
- [Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation](https://mlsanity.com/paper/2301.03396/) (2023)

## Built on by (most cited first, 1 of 1)

- [Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework](https://mlsanity.com/paper/2511.08613/) (2025)

## Abstract (arXiv)

Audio-Driven Talking Face Generation aims at generating realistic videos of talking faces, focusing on accurate audio-lip synchronization without deteriorating any identity-related visual details. Recent state-of-the-art methods are based on inpainting, meaning that the lower half of the input face is masked, and the model fills the masked region by generating lips aligned with the given audio. Hence, to preserve identity-related visual details from the lower half, these approaches additionally require an unmasked identity reference image randomly selected from the same video. However, this common masking strategy suffers from (1) information loss in the input faces, significantly affecting the networks' ability to preserve visual quality and identity details, (2) variation between identity reference and input image degrading reconstruction performance, and (3) the identity reference negatively impacting the model, causing unintended copying of elements unaligned with the audio. To address these issues, we propose a mask-free talking face generation approach while maintaining the 2D-based face editing task. Instead of masking the lower half, we transform the input images to have closed mouths, using a two-step landmark-based approach trained in an unpaired manner. Subsequently, we provide these edited but unmasked faces to a lip adaptation model alongside the audio to generate appropriate lip movements. Thus, our approach needs neither masked input images nor identity reference images. We conduct experiments on the benchmark LRS2 and HDTF datasets and perform various ablation studies to validate our contributions.