# UniFLG: Unified Facial Landmark Generator from Text or Speech

arXiv: https://arxiv.org/abs/2302.14337 · HTML: https://mlsanity.com/paper/2302.14337/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 25 of 25)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech](https://mlsanity.com/paper/2106.06103/) (2021)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Audio-Driven Emotional Video Portraits](https://mlsanity.com/paper/2104.07452/) (2021)
- [Deep Audio-Visual Learning: A Survey](https://mlsanity.com/paper/2001.04758/) (2020)
- [Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation](https://mlsanity.com/paper/2104.07995/) (2021)
- [ObamaNet: Photo-realistic lip-sync from text](https://mlsanity.com/paper/1801.01442/) (2017)
- [Text2Video: Text-driven Talking-head Video Synthesis with Personalized Phoneme-Pose Dictionary](https://mlsanity.com/paper/2104.14631/) (2021)
- [AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person](https://mlsanity.com/paper/2108.04325/) (2021)
- [Generating Talking Face Landmarks from Speech](https://mlsanity.com/paper/1803.09803/) (2018)
- [Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation](https://mlsanity.com/paper/2112.02214/) (2021)
- [Investigation of enhanced Tacotron text-to-speech synthesis systems with self-attention for pitch accent language](https://mlsanity.com/paper/1810.11960/) (2018)
- [Unsupervised Audiovisual Synthesis via Exemplar Autoencoders](https://mlsanity.com/paper/2001.04463/) (2020)
- [Audiovisual Speech Synthesis using Tacotron2](https://mlsanity.com/paper/2008.00620/) (2020)
- [End-to-End Text-to-Speech Based on Latent Representation of Speaking Styles Using Spontaneous Dialogue](https://mlsanity.com/paper/2206.12040/) (2022)

## Built on by (most cited first, 8 of 8)

- [Faces that Speak: Jointly Synthesising Talking Face and Speech from Text](https://mlsanity.com/paper/2405.10272/) (2024)
- [AV-Flow: Transforming Text to Audio-Visual Human-like Interactions](https://mlsanity.com/paper/2502.13133/) (2025)
- [OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking](https://mlsanity.com/paper/2504.02433/) (2025)
- [Unified speech and gesture synthesis using flow matching](https://mlsanity.com/paper/2310.05181/) (2023)
- [Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism](https://mlsanity.com/paper/2312.06613/) (2023)
- [Face2VoiceSync: Lightweight Face-Voice Consistency for Text-Driven Talking Face Generation](https://mlsanity.com/paper/2507.19225/) (2025)
- [Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis](https://mlsanity.com/paper/2511.05432/) (2025)
- [Text-Driven Emotionally Continuous Talking Face Generation](https://mlsanity.com/paper/2603.06071/) (2026)

## Abstract (arXiv)

Talking face generation has been extensively investigated owing to its wide applicability. The two primary frameworks used for talking face generation comprise a text-driven framework, which generates synchronized speech and talking faces from text, and a speech-driven framework, which generates talking faces from speech. To integrate these frameworks, this paper proposes a unified facial landmark generator (UniFLG). The proposed system exploits end-to-end text-to-speech not only for synthesizing speech but also for extracting a series of latent representations that are common to text and speech, and feeds it to a landmark decoder to generate facial landmarks. We demonstrate that our system achieves higher naturalness in both speech synthesis and facial landmark generation compared to the state-of-the-art text-driven method. We further demonstrate that our system can generate facial landmarks from speech of speakers without facial video data or even speech data.