# Talking Head Synthesis with Facial Landmark Guidance via 3D Gaussian Splatting

arXiv: https://arxiv.org/abs/2609.17422 · HTML: https://mlsanity.com/paper/2609.17422/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Builds on (most cited first, 30 of 37)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://mlsanity.com/paper/2008.10010/) (2020)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Sapiens: Foundation for Human Vision Models](https://mlsanity.com/paper/2408.12569/) (2024)
- [GaussianPro: 3D Gaussian Splatting with Progressive Propagation](https://mlsanity.com/paper/2402.14650/) (2024)
- [Lip Movements Generation at a Glance](https://mlsanity.com/paper/1803.10404/) (2018)
- [GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis](https://mlsanity.com/paper/2301.13430/) (2023)
- [Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation](https://mlsanity.com/paper/2201.07786/) (2022)
- [Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks](https://mlsanity.com/paper/2105.02358/) (2021)
- [Identity-Preserving Talking Face Generation with Landmark and Appearance Priors](https://mlsanity.com/paper/2305.08293/) (2023)
- [SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis](https://mlsanity.com/paper/2311.17590/) (2023)
- [Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition](https://mlsanity.com/paper/2211.12368/) (2022)
- [Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis](https://mlsanity.com/paper/2307.09323/) (2023)
- [One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field](https://mlsanity.com/paper/2304.05097/) (2023)
- [TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting](https://mlsanity.com/paper/2404.15264/) (2024)
- [Sound Adversarial Audio-Visual Navigation](https://mlsanity.com/paper/2202.10910/) (2022)
- [FLNet: Landmark Driven Fetching and Learning Network for Faithful Talking Facial Animation Synthesis](https://mlsanity.com/paper/1911.09224/) (2019)
- [Measuring Acoustics with Collaborative Multiple Agents](https://mlsanity.com/paper/2310.05368/) (2023)
- [InsTaG: Learning Personalized 3D Talking Head from Few-Second Video](https://mlsanity.com/paper/2502.20387/) (2025)
- [GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splatting](https://mlsanity.com/paper/2404.16012/) (2024)
- [Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation](https://mlsanity.com/paper/2509.16924/) (2025)
- [DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigation](https://mlsanity.com/paper/2505.00743/) (2025)
- [Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environments](https://mlsanity.com/paper/2509.22698/) (2025)
- [Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Tasks](https://mlsanity.com/paper/2509.25652/) (2025)

## Abstract (arXiv)

Audio-driven digital human generation plays an important role in virtual communication, immersive interaction, and media production. With the development of Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), recent talking-head systems have obtained more faithful 3D facial geometry and appearance modeling. A remaining difficulty is that speech features mainly describe temporal acoustic patterns rather than explicit facial layouts. As a result, directly driving 3D facial deformation with audio may produce inaccurate mouth motion, weak expression details, and local artifacts. To address this issue, we propose a facial-keypoint-guided spatial enhancement module. The predicted landmarks provide structural cues for selecting and enriching spatial points around expression-sensitive facial regions. We further introduce a global landmark compensation mechanism, where the full set of keypoints is encoded into a conditioning vector to refine 3DGS attributes. This compensation supplies whole-face structural information to the underlying shape representation. Experiments under self-driven and cross-driven settings show that the proposed method improves visual quality, facial realism, and lip synchronization.