# Text2Video: Text-driven Talking-head Video Synthesis with Personalized Phoneme-Pose Dictionary

arXiv: https://arxiv.org/abs/2104.14631 · HTML: https://mlsanity.com/paper/2104.14631/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 29 Apr 2021.

## Builds on (most cited first, 7 of 7)

- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [Talking Face Generation by Adversarially Disentangled Audio-Visual Representation](https://mlsanity.com/paper/1807.07860/) (2018)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [Text-based Editing of Talking-head Video](https://mlsanity.com/paper/1906.01524/) (2019)

## Built on by (most cited first, 27 of 27)

- [SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces](https://mlsanity.com/paper/2306.10799/) (2023)
- [FT2TF: First-Person Statement Text-To-Talking Face Generation](https://mlsanity.com/paper/2312.05430/) (2023)
- [SignDiff: Diffusion Model for American Sign Language Production](https://mlsanity.com/paper/2308.16082/) (2023)
- [Faces that Speak: Jointly Synthesising Talking Face and Speech from Text](https://mlsanity.com/paper/2405.10272/) (2024)
- [AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person](https://mlsanity.com/paper/2108.04325/) (2021)
- [Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlook](https://mlsanity.com/paper/2411.19537/) (2024)
- [DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generation](https://mlsanity.com/paper/2410.07151/) (2024)
- [AV-Flow: Transforming Text to Audio-Visual Human-like Interactions](https://mlsanity.com/paper/2502.13133/) (2025)
- [OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking](https://mlsanity.com/paper/2504.02433/) (2025)
- [UniFLG: Unified Facial Landmark Generator from Text or Speech](https://mlsanity.com/paper/2302.14337/) (2023)
- [Text-to-Video: a Two-stage Framework for Zero-shot Identity-agnostic Talking-head Generation](https://mlsanity.com/paper/2308.06457/) (2023)
- [GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT](https://mlsanity.com/paper/2303.13013/) (2023)
- [Text-driven Talking Face Synthesis by Reprogramming Audio-driven Models](https://mlsanity.com/paper/2306.16003/) (2023)
- [Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions](https://mlsanity.com/paper/2507.02900/) (2025)
- [DATA: Multi-Disentanglement based Contrastive Learning for Open-World Semi-Supervised Deepfake Attribution](https://mlsanity.com/paper/2505.04384/) (2025)
- [Synthesizing Photorealistic Virtual Humans Through Cross-modal Disentanglement](https://mlsanity.com/paper/2209.01320/) (2022)
- [Unsupervised Sign Language Translation and Generation](https://mlsanity.com/paper/2402.07726/) (2024)
- [Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism](https://mlsanity.com/paper/2312.06613/) (2023)
- [SignLLM: Sign Language Production Large Language Models](https://mlsanity.com/paper/2405.10718/) (2024)
- [SARGes: Semantically Aligned Reliable Gesture Generation via Intent Chain](https://mlsanity.com/paper/2503.20202/) (2025)
- [FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency](https://mlsanity.com/paper/2504.04427/) (2025)
- [NLDF: Neural Light Dynamic Fields for Efficient 3D Talking Head Generation](https://mlsanity.com/paper/2406.11259/) (2024)
- [3D Engine-ready Photorealistic Avatars via Dynamic Textures](https://mlsanity.com/paper/2503.14943/) (2025)
- [Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation](https://mlsanity.com/paper/2507.19225/) (2025)
- [Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis](https://mlsanity.com/paper/2511.05432/) (2025)
- [Narrating For You: Prompt-guided Audio-visual Narrating Face Generation Employing Multi-entangled Latent Space](https://mlsanity.com/paper/2602.18618/) (2026)
- [Text-Driven Emotionally Continuous Talking Face Generation](https://mlsanity.com/paper/2603.06071/) (2026)

## Abstract (arXiv)

With the advance of deep learning technology, automatic video generation from audio or text has become an emerging and promising research topic. In this paper, we present a novel approach to synthesize video from the text. The method builds a phoneme-pose dictionary and trains a generative adversarial network (GAN) to generate video from interpolated phoneme poses. Compared to audio-driven video generation algorithms, our approach has a number of advantages: 1) It only needs a fraction of the training data used by an audio-driven approach; 2) It is more flexible and not subject to vulnerability due to speaker variation; 3) It significantly reduces the preprocessing, training and inference time. We perform extensive experiments to compare the proposed method with state-of-the-art talking face generation methods on a benchmark dataset and datasets of our own. The results demonstrate the effectiveness and superiority of our approach.