# OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions arXiv: https://arxiv.org/abs/2309.16148 ยท HTML: https://mlsanity.com/paper/2309.16148/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 57 of its 81 references that are in the map. - First posted on arXiv 28 Sep 2023. ## Builds on (most cited first, 30 of 57) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018) - [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016) - [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019) - [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017) - [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017) - [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018) - [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020) - [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020) - [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020) - [How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)](https://mlsanity.com/paper/1703.07332/) (2017) - [Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set](https://mlsanity.com/paper/1903.08527/) (2019) - [One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing](https://mlsanity.com/paper/2011.15126/) (2020) - [Tutorial on Variational Autoencoders](https://mlsanity.com/paper/1606.05908/) (2016) - [SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation](https://mlsanity.com/paper/2211.12194/) (2022) - [Memory Networks](https://mlsanity.com/paper/1410.3916/) (2014) - [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021) - [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020) - [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021) - [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019) - [Deep Video Portraits](https://mlsanity.com/paper/1805.11714/) (2018) - [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019) - [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020) - [Key-Value Memory Networks for Directly Reading Documents](https://mlsanity.com/paper/1606.03126/) (2016) - [Talking Face Generation by Adversarially Disentangled Audio-Visual Representation](https://mlsanity.com/paper/1807.07860/) (2018) - [Thin-Plate Spline Motion Model for Image Animation](https://mlsanity.com/paper/2203.14367/) (2022) - [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018) - [PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering](https://mlsanity.com/paper/2109.08379/) (2021) - [Depth-Aware Generative Adversarial Network for Talking Head Video Generation](https://mlsanity.com/paper/2203.06605/) (2022) - [Fine-Grained Head Pose Estimation Without Keypoints](https://mlsanity.com/paper/1710.00925/) (2017) ## Built on by (most cited first, 5 of 5) - [SingingHead: A Large-scale 4D Dataset for Singing Head Animation](https://mlsanity.com/paper/2312.04369/) (2023) - [Training-Free Video Editing via Optical Flow-Enhanced Score Distillation](https://mlsanity.com/paper/2406.04888/) (2024) - [M2DAO-Talker: Harmonizing Multi-granular Motion Decoupling and Alternating Optimization for Talking-head Generation](https://mlsanity.com/paper/2507.08307/) (2025) - [High-Fidelity 3D Facial Avatar Synthesis with Controllable Fine-Grained Expressions](https://mlsanity.com/paper/2603.14781/) (2026) - [PAGE: Parts-Aware Guidance for Co-Speech Gesture Portrait Video Generation](https://mlsanity.com/paper/2610.13301/) (2026) ## Abstract (arXiv) One-shot talking head generation has no explicit head movement reference, thus it is difficult to generate talking heads with head motions. Some existing works only edit the mouth area and generate still talking heads, leading to unreal talking head performance. Other works construct one-to-one mapping between audio signal and head motion sequences, introducing ambiguity correspondences into the mapping since people can behave differently in head motions when speaking the same content. This unreasonable mapping form fails to model the diversity and produces either nearly static or even exaggerated head motions, which are unnatural and strange. Therefore, the one-shot talking head generation task is actually a one-to-many ill-posed problem and people present diverse head motions when speaking. Based on the above observation, we propose OSM-Net, a textitone-to-many one-shot talking head generation network with natural head motions. OSM-Net constructs a motion space that contains rich and various clip-level head motion features. Each basis of the space represents a feature of meaningful head motion in a clip rather than just a frame, thus providing more coherent and natural motion changes in talking heads. The driving audio is mapped into the motion space, around which various motion features can be sampled within a reasonable range to achieve the one-to-many mapping. Besides, the landmark constraint and time window feature input improve the accurate expression feature extraction and video generation. Extensive experiments show that OSM-Net generates more natural realistic head motions under reasonable one-to-many mapping paradigm compared with other methods.