# FONT: Flow-guided One-shot Talking Head Generation with Natural Head Motions

arXiv: https://arxiv.org/abs/2303.17789 · HTML: https://mlsanity.com/paper/2303.17789/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019)
- [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Talking Face Generation by Adversarially Disentangled Audio-Visual Representation](https://mlsanity.com/paper/1807.07860/) (2018)
- [StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN](https://mlsanity.com/paper/2203.04036/) (2022)
- [Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion](https://mlsanity.com/paper/2107.09293/) (2021)
- [Talking-head Generation with Rhythmic Head Motion](https://mlsanity.com/paper/2007.08547/) (2020)
- [You said that?](https://mlsanity.com/paper/1705.02966/) (2017)
- [Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesis](https://mlsanity.com/paper/2211.14506/) (2022)
- [One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning](https://mlsanity.com/paper/2112.02749/) (2021)
- [SPACE: Speech-driven Portrait Animation with Controllable Expression](https://mlsanity.com/paper/2211.09809/) (2022)
- [What comprises a good talking-head video generation?: A Survey and Benchmark](https://mlsanity.com/paper/2005.03201/) (2020)

## Built on by (most cited first, 5 of 5)

- [MFR-Net: Multi-faceted Responsive Listening Head Generation via Denoising Diffusion Model](https://mlsanity.com/paper/2308.16635/) (2023)
- [OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions](https://mlsanity.com/paper/2309.16148/) (2023)
- [A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editing](https://mlsanity.com/paper/2406.10553/) (2024)
- [Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation](https://mlsanity.com/paper/2507.20953/) (2025)
- [G4G:A Generic Framework for High Fidelity Talking Face Generation with Fine-grained Intra-modal Alignment](https://mlsanity.com/paper/2402.18122/) (2024)

## Abstract (arXiv)

One-shot talking head generation has received growing attention in recent years, with various creative and practical applications. An ideal natural and vivid generated talking head video should contain natural head pose changes. However, it is challenging to map head pose sequences from driving audio since there exists a natural gap between audio-visual modalities. In this work, we propose a Flow-guided One-shot model that achieves NaTural head motions(FONT) over generated talking heads. Specifically, the head pose prediction module is designed to generate head pose sequences from the source face and driving audio. We add the random sampling operation and the structural similarity constraint to model the diversity in the one-to-many mapping between audio-visual modality, thus predicting natural head poses. Then we develop a keypoint predictor that produces unsupervised keypoints from the source face, driving audio and pose sequences to describe the facial structure information. Finally, a flow-guided occlusion-aware generator is employed to produce photo-realistic talking head videos from the estimated keypoints and source face. Extensive experimental results prove that FONT generates talking heads with natural head poses and synchronized mouth shapes, outperforming other compared methods.