mlsanityData to 14 Oct 2026

Talking Head Synthesis with Facial Landmark Guidance via 3D Gaussian Splatting

arXiv 2609.17422 ↗First posted 16 Jul 2026cs.AI eess.SPcited by 0

Not on the map. Why?

Abstract (arXiv)

Audio-driven digital human generation plays an important role in virtual communication, immersive interaction, and media production. With the development of Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), recent talking-head systems have obtained more faithful 3D facial geometry and appearance modeling. A remaining difficulty is that speech features mainly describe temporal acoustic patterns rather than explicit facial layouts. As a result, directly driving 3D facial deformation with audio may produce inaccurate mouth motion, weak expression details, and local artifacts. To address this issue, we propose a facial-keypoint-guided spatial enhancement module. The predicted landmarks provide structural cues for selecting and enriching spatial points around expression-sensitive facial regions. We further introduce a global landmark compensation mechanism, where the full set of keypoints is encoded into a conditioning vector to refine 3DGS attributes. This compensation supplies whole-face structural information to the underlying shape representation. Experiments under self-driven and cross-driven settings show that the proposed method improves visual quality, facial realism, and lip synchronization.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

37 of 47 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  4. 2017Squeeze-and-Excitation Networkscited by 4,789
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  7. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  8. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  9. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  10. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  11. 2024GaussianPro: 3D Gaussian Splatting with Progressive Propagationcited by 200
  12. 2018Lip Movements Generation at a Glancecited by 155
25 more
  1. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  2. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  3. 2021Beyond Self-attention: External Attention using Two Linear Layers for Visual Taskscited by 109
  4. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 98
  5. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  6. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  7. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72
  8. 2023One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Fieldcited by 68
  9. 2024TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splattingcited by 60
  10. 2022Sound Adversarial Audio-Visual Navigationcited by 48
  11. 2019FLNet: Landmark Driven Fetching and Learning Network for Faithful Talking Facial Animation Synthesiscited by 28
  12. 2023Measuring Acoustics with Collaborative Multiple Agentscited by 22
  13. 2025InsTaG: Learning Personalized 3D Talking Head from Few-Second Videocited by 18
  14. 2024GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splattingcited by 17
  15. 2025Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigationcited by 15
  16. 2025DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigationcited by 14
  17. 2025Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environmentscited by 14
  18. 2025Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Taskscited by 14
  19. 2025DGFNet: End-to-End Audio-Visual Source Separation Based on Dynamic Gating Fusioncited by 13
  20. 2025Dynamic Multi-Target Fusion for Efficient Audio-Visual Navigationcited by 13
  21. 2024BSS-CFFMA: Cross-Domain Feature Fusion and Multi-Attention Speech Enhancement Network based on Self-Supervised Embeddingcited by 10
  22. 2024VNet: A GAN-based Multi-Tier Discriminator Network for Speech Synthesis Vocoderscited by 10
  23. 2025Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysiscited by 8
  24. 2024DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesiscited by 7
  25. 2025FSDENet: A Frequency and Spatial Domains based Detail Enhancement Network for Remote Sensing Semantic Segmentationcited by 7

Built on by

Not cited yet.