# Assessing Identity Leakage in Talking Face Generation: Metrics and Evaluation Framework

arXiv: https://arxiv.org/abs/2511.08613 · HTML: https://mlsanity.com/paper/2511.08613/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 5 Nov 2025.

## Builds on (most cited first, 26 of 26)

- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions](https://mlsanity.com/paper/2402.17485/) (2024)
- [Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation](https://mlsanity.com/paper/2301.03396/) (2023)
- [VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild](https://mlsanity.com/paper/2211.14758/) (2022)
- [Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert](https://mlsanity.com/paper/2303.17480/) (2023)
- [Identity-Preserving Talking Face Generation with Landmark and Appearance Priors](https://mlsanity.com/paper/2305.08293/) (2023)
- [Everybody's Talkin': Let Me Talk as You Want](https://mlsanity.com/paper/2001.05201/) (2020)
- [DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video](https://mlsanity.com/paper/2303.03988/) (2023)
- [SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory](https://mlsanity.com/paper/2211.00924/) (2022)
- [LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision](https://mlsanity.com/paper/2412.09262/) (2024)
- [Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronization](https://mlsanity.com/paper/2308.09716/) (2023)
- [Low-Latency Neural Speech Translation](https://mlsanity.com/paper/1808.00491/) (2018)
- [MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Sampling](https://mlsanity.com/paper/2410.10122/) (2024)
- [OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers](https://mlsanity.com/paper/2505.21448/) (2025)
- [Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videos](https://mlsanity.com/paper/2206.04523/) (2022)
- [Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation](https://mlsanity.com/paper/2405.04327/) (2024)
- [Audio-driven Talking Face Generation with Stabilized Synchronization Loss](https://mlsanity.com/paper/2307.09368/) (2023)
- [EmoTalker: Emotionally Editable Talking Face Generation via Diffusion Model](https://mlsanity.com/paper/2401.08049/) (2024)
- [KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution](https://mlsanity.com/paper/2505.00497/) (2025)
- [EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion](https://mlsanity.com/paper/2409.07255/) (2024)
- [Titanic Calling: Low Bandwidth Video Conference from the Titanic Wreck](https://mlsanity.com/paper/2410.11434/) (2024)
- [Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation](https://mlsanity.com/paper/2507.20953/) (2025)

## Built on by (most cited first, 1 of 1)

- [CIViC: Evidence Gates for Audio-Conditioned Visual Prediction on Proxy Tasks](https://mlsanity.com/paper/2610.15327/) (2026)

## Abstract (arXiv)

Video editing-based talking face generation aims to preserve video details such as pose, lighting, and gestures while modifying only lip motion, often using an identity reference image to maintain speaker consistency. However, this mechanism can introduce lip leakage, where generated lips are influenced by the reference image rather than solely by the driving audio. Such leakage is difficult to detect with standard metrics and conventional test setup. To address this, we propose a systematic evaluation methodology to analyze and quantify lip leakage. Our framework employs three complementary test setups: silent-input generation, mismatched audio-video pairing, and matched audio-video synthesis. We also introduce derived metrics including lip-sync discrepancy and silent-audio-based lip-sync scores. In addition, we study how different identity reference selections affect leakage, providing insights into reference design. The proposed methodology is model-agnostic and establishes a more reliable benchmark for future research in talking face generation.