SDTalk: Structured Facial Priors and Dual-Branch Motion Fields for Generalizable Gaussian Talking Head Synthesis
Not on the map. Why?
Abstract (arXiv)
High-quality, real-time talking head synthesis remains a fundamental challenge in computer vision. Existing reconstruction- and rendering-based methods typically rely on identity-specific models, limiting cross-identity generalization. To address this issue, we propose SDTalk, a one-shot 3D Gaussian Splatting (3DGS)-based framework that generalizes to unseen identities without personalized training or fine-tuning. Our framework comprises two modules with a two-stage training strategy. In the first stage, we incorporate structured facial priors into the reconstruction module and separately predict 3DGS parameters for visible and occluded regions, enabling complete head reconstruction from a single image. In the second stage, we introduce a dual-branch motion field to model coarse and fine facial dynamics, improving detail fidelity and lip synchronization. Experiments demonstrate that SDTalk surpasses existing methods in both visual quality and inference efficiency.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 11 May 2026.
Builds on
18 of 24 references.
References · 18
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
- 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
- 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
- 2018Lip Movements Generation at a Glancecited by 155
- 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animationcited by 136
- 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
- 2024Generalizable and Animatable Gaussian Head Avatarcited by 97
- 2023Efficient Emotional Adaptation for Audio-Driven Talking-Head Generationcited by 81
6 more
- 2024TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splattingcited by 60
- 2025InsTaG: Learning Personalized 3D Talking Head from Few-Second Videocited by 18
- 2024MimicTalk: Mimicking a personalized and expressive 3D talking face in minutescited by 16
- 2024NeRFFaceSpeech: One-shot Audio-driven 3D Talking Head Synthesis via Generative Priorcited by 8
- 2024DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesiscited by 7
- 2025NeRF-3DTalker: Neural Radiance Field with 3D Prior Aided Audio Disentanglement for Talking Head Synthesiscited by 3
Built on by
Not cited yet.