# What comprises a good talking-head video generation?: A Survey and Benchmark

arXiv: https://arxiv.org/abs/2005.03201 · HTML: https://mlsanity.com/paper/2005.03201/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2020.

## Built on by (most cited first, 30 of 42)

- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis](https://mlsanity.com/paper/2401.08503/) (2024)
- [StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles](https://mlsanity.com/paper/2301.01081/) (2023)
- [StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generator](https://mlsanity.com/paper/2305.05445/) (2023)
- [DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering](https://mlsanity.com/paper/2201.00791/) (2022)
- [GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation](https://mlsanity.com/paper/2305.00787/) (2023)
- [A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights](https://mlsanity.com/paper/2407.08428/) (2024)
- [Towards Realistic Visual Dubbing with Heterogeneous Sources](https://mlsanity.com/paper/2201.06260/) (2022)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [Faces that Speak: Jointly Synthesising Talking Face and Speech from Text](https://mlsanity.com/paper/2405.10272/) (2024)
- [TALK-Act: Enhance Textural-Awareness for 2D Speaking Avatar Reenactment with Diffusion Model](https://mlsanity.com/paper/2410.10696/) (2024)
- [AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person](https://mlsanity.com/paper/2108.04325/) (2021)
- [Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis](https://mlsanity.com/paper/2109.02081/) (2021)
- [MFR-Net: Multi-faceted Responsive Listening Head Generation via Denoising Diffusion Model](https://mlsanity.com/paper/2308.16635/) (2023)
- [MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes](https://mlsanity.com/paper/2410.06734/) (2024)
- [Continuously Controllable Facial Expression Editing in Talking Face Videos](https://mlsanity.com/paper/2209.08289/) (2022)
- [Freeform Body Motion Generation from Speech](https://mlsanity.com/paper/2203.02291/) (2022)
- [Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video](https://mlsanity.com/paper/2309.04814/) (2023)
- [StyleTalk++: A Unified Framework for Controlling the Speaking Styles of Talking Heads](https://mlsanity.com/paper/2409.09292/) (2024)
- [KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation](https://mlsanity.com/paper/2503.01715/) (2025)
- [From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applications](https://mlsanity.com/paper/2308.16041/) (2023)
- [That's What I Said: Fully-Controllable Talking Face Generation](https://mlsanity.com/paper/2304.03275/) (2023)
- [Make Your Actor Talk: Generalizable and High-Fidelity Lip Sync with Motion and Appearance Disentanglement](https://mlsanity.com/paper/2406.08096/) (2024)
- [UniSync: A Unified Framework for Audio-Visual Synchronization](https://mlsanity.com/paper/2503.16357/) (2025)
- [FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motions](https://mlsanity.com/paper/2303.17789/) (2023)
- [OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions](https://mlsanity.com/paper/2309.16148/) (2023)

## Abstract (arXiv)

Over the years, performance evaluation has become essential in computer vision, enabling tangible progress in many sub-fields. While talking-head video generation has become an emerging research topic, existing evaluations on this topic present many limitations. For example, most approaches use human subjects (e.g., via Amazon MTurk) to evaluate their research claims directly. This subjective evaluation is cumbersome, unreproducible, and may impend the evolution of new research. In this work, we present a carefully-designed benchmark for evaluating talking-head video generation with standardized dataset pre-processing strategies. As for evaluation, we either propose new metrics or select the most appropriate ones to evaluate results in what we consider as desired properties for a good talking-head video, namely, identity preserving, lip synchronization, high video quality, and natural-spontaneous motion. By conducting a thoughtful analysis across several state-of-the-art talking-head generation approaches, we aim to uncover the merits and drawbacks of current methods and point out promising directions for future work. All the evaluation code is available at: https://github.com/lelechen63/talking-head-generation-survey.