# Text-to-Video: a Two-stage Framework for Zero-shot Identity-agnostic Talking-head Generation

arXiv: https://arxiv.org/abs/2308.06457 · HTML: https://mlsanity.com/paper/2308.06457/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 18 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers](https://mlsanity.com/paper/2301.02111/) (2023)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone](https://mlsanity.com/paper/2112.02418/) (2021)
- [StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN](https://mlsanity.com/paper/2203.04036/) (2022)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild](https://mlsanity.com/paper/2211.14758/) (2022)
- [One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning](https://mlsanity.com/paper/2112.02749/) (2021)
- [Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition](https://mlsanity.com/paper/2211.12368/) (2022)
- [End-to-End Text-Dependent Speaker Verification](https://mlsanity.com/paper/1509.08062/) (2015)
- [Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation](https://mlsanity.com/paper/2104.07995/) (2021)
- [ObamaNet: Photo-realistic lip-sync from text](https://mlsanity.com/paper/1801.01442/) (2017)
- [Text2Video: Text-driven Talking-head Video Synthesis with Personalized Phoneme-Pose Dictionary](https://mlsanity.com/paper/2104.14631/) (2021)
- [Memories are One-to-Many Mapping Alleviators in Talking Face Generation](https://mlsanity.com/paper/2212.05005/) (2022)

## Built on by (most cited first, 8 of 8)

- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [Faces that Speak: Jointly Synthesising Talking Face and Speech from Text](https://mlsanity.com/paper/2405.10272/) (2024)
- [DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generation](https://mlsanity.com/paper/2410.07151/) (2024)
- [Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions](https://mlsanity.com/paper/2507.02900/) (2025)
- [Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation](https://mlsanity.com/paper/2507.19225/) (2025)
- [Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis](https://mlsanity.com/paper/2511.05432/) (2025)
- [Narrating For You: Prompt-guided Audio-visual Narrating Face Generation Employing Multi-entangled Latent Space](https://mlsanity.com/paper/2602.18618/) (2026)
- [Text-Driven Emotionally Continuous Talking Face Generation](https://mlsanity.com/paper/2603.06071/) (2026)

## Abstract (arXiv)

The advent of ChatGPT has introduced innovative methods for information gathering and analysis. However, the information provided by ChatGPT is limited to text, and the visualization of this information remains constrained. Previous research has explored zero-shot text-to-video (TTV) approaches to transform text into videos. However, these methods lacked control over the identity of the generated audio, i.e., not identity-agnostic, hindering their effectiveness. To address this limitation, we propose a novel two-stage framework for person-agnostic video cloning, specifically focusing on TTV generation. In the first stage, we leverage pretrained zero-shot models to achieve text-to-speech (TTS) conversion. In the second stage, an audio-driven talking head generation method is employed to produce compelling videos privided the audio generated in the first stage. This paper presents a comparative analysis of different TTS and audio-driven talking head generation methods, identifying the most promising approach for future research and development. Some audio and videos samples can be found in the following link: https://github.com/ZhichaoWang970201/Text-to-Video/tree/main.