# InsTaG: Learning Personalized 3D Talking Head from Few-Second Video

arXiv: https://arxiv.org/abs/2502.20387 · HTML: https://mlsanity.com/paper/2502.20387/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 71 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 30 of 51)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction](https://mlsanity.com/paper/2012.03065/) (2020)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation](https://mlsanity.com/paper/2104.11116/) (2021)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization](https://mlsanity.com/paper/2403.06912/) (2024)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Neural Voice Puppetry: Audio-driven Facial Reenactment](https://mlsanity.com/paper/1912.05566/) (2019)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [HeadNeRF: A Real-time NeRF-based Parametric Head Model](https://mlsanity.com/paper/2112.05637/) (2021)
- [GaussianPro: 3D Gaussian Splatting with Progressive Propagation](https://mlsanity.com/paper/2402.14650/) (2024)
- [FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction](https://mlsanity.com/paper/2003.13989/) (2020)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time](https://mlsanity.com/paper/2404.10667/) (2024)
- [Lip Movements Generation at a Glance](https://mlsanity.com/paper/1803.10404/) (2018)
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](https://mlsanity.com/paper/1902.05978/) (2019)
- [GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis](https://mlsanity.com/paper/2301.13430/) (2023)
- [Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation](https://mlsanity.com/paper/2201.07786/) (2022)
- [Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis](https://mlsanity.com/paper/2207.11770/) (2022)
- [Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis](https://mlsanity.com/paper/2401.08503/) (2024)
- [Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose](https://mlsanity.com/paper/2002.10137/) (2020)
- [FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning](https://mlsanity.com/paper/2108.07938/) (2021)

## Built on by (most cited first, 18 of 18)

- [GGTalker: Talking Head Systhesis with Generalizable Gaussian Priors and Identity-Specific Adaptation](https://mlsanity.com/paper/2506.21513/) (2025)
- [Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics](https://mlsanity.com/paper/2512.15340/) (2025)
- [EmoTaG: Emotion-Aware Talking Head Synthesis on Gaussian Splatting with Few-Shot Personalization](https://mlsanity.com/paper/2603.21332/) (2026)
- [FaceEditTalker: Controllable Talking Head Generation with Facial Attribute Editing](https://mlsanity.com/paper/2505.22141/) (2025)
- [Few-Shot Identity Adaptation for 3D Talking Heads via Global Gaussian Field](https://mlsanity.com/paper/2506.22044/) (2025)
- [ISExplore:Informative Segment Selection for Efficient Personalized 3D Talking Face Generation](https://mlsanity.com/paper/2511.07940/) (2025)
- [EmoDiffTalk:Emotion-aware Diffusion for Editable 3D Gaussian Talking Head](https://mlsanity.com/paper/2512.05991/) (2025)
- [Efficient and Robust Video Defense Framework against 3D-field Personalized Talking Face](https://mlsanity.com/paper/2512.21019/) (2025)
- [Toward Fine-Grained Facial Control in 3D Talking Head Generation](https://mlsanity.com/paper/2602.09736/) (2026)
- [Low-Rank Head Avatar Personalization with Registers](https://mlsanity.com/paper/2506.01935/) (2025)
- [HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis](https://mlsanity.com/paper/2508.10566/) (2025)
- [3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars](https://mlsanity.com/paper/2602.10516/) (2026)
- [EmbedTalk: Talking Head Synthesis using Gaussian Embeddings](https://mlsanity.com/paper/2603.07604/) (2026)
- [SDTalk: Structured Facial Priors and Dual-Branch Motion Fields for Generalizable Gaussian Talking Head Synthesis](https://mlsanity.com/paper/2605.09956/) (2026)
- [Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation](https://mlsanity.com/paper/2606.01031/) (2026)
- [FlexiAvatar: Unified 3D Gaussian Human Avatars Under Arbitrary Body Visibility](https://mlsanity.com/paper/2607.19100/) (2026)
- [Audio-Driven Adversarial Defense for 3D Talking Face Generation with totally Visual Fidelity Preservation](https://mlsanity.com/paper/2608.30951/) (2026)
- [Talking Head Synthesis with Facial Landmark Guidance via 3D Gaussian Splatting](https://mlsanity.com/paper/2609.17422/) (2026)

## Abstract (arXiv)

Despite exhibiting impressive performance in synthesizing lifelike personalized 3D talking heads, prevailing methods based on radiance fields suffer from high demands for training data and time for each new identity. This paper introduces InsTaG, a 3D talking head synthesis framework that allows a fast learning of realistic personalized 3D talking head from few training data. Built upon a lightweight 3DGS person-specific synthesizer with universal motion priors, InsTaG achieves high-quality and fast adaptation while preserving high-level personalization and efficiency. As preparation, we first propose an Identity-Free Pre-training strategy that enables the pre-training of the person-specific model and encourages the collection of universal motion priors from long-video data corpus. To fully exploit the universal motion priors to learn an unseen new identity, we then present a Motion-Aligned Adaptation strategy to adaptively align the target head to the pre-trained field, and constrain a robust dynamic head structure under few training data. Experiments demonstrate our outstanding performance and efficiency under various data scenarios to render high-quality personalized talking heads.