mlsanityData to 14 Oct 2026

PD-GS: Phoneme-Driven 3DGS for Audio-Driven Talking Heads

arXiv 2608.05218 ↗First posted 5 Aug 2026cs.AI cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

3D Gaussian Splatting (3DGS) enables fast, photorealistic talking-head rendering, yet accurate lip articulation remains elusive: mouth motion is often over-smoothed and may violate hard articulatory constraints such as bilabial closures, producing the notorious ``leaky mouth'' artifact. A key difficulty is that brief, discrete articulatory events are inferred from a continuous acoustic embedding under a regression objective, which biases predictions toward averaged mouth configurations. While modern self-supervised speech encoders provide rich prosodic and phonetic cues, they do not provide an explicit, frame-aligned linguistic target that reliably disambiguates closure-level events. We propose Phoneme-Driven Gaussian Splatting (PD-GS), which augments a 3DGS talker with time-aligned phoneme tokens obtained from an automatic ASR and forced-alignment pipeline. Our core component, the Linguistic Fusion Module (LFM), adaptively fuses continuous audio context with discrete phoneme embeddings through a learned gate, allowing the model to preserve smooth audio-driven dynamics while strengthening phoneme guidance on articulation-critical segments. PD-GS is trained purely from monocular video using image reconstruction and lip landmark supervision. On HDTF, PD-GS achieves the best lip geometry among the compared baselines (LMD 2.66) and qualitatively reduces closure violations in challenging phoneme sequences, yielding more linguistically faithful neural avatars.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 35 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

23 of 35 references.

References · 23

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  5. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  6. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  7. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 259
  8. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 184
  9. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  10. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 122
  11. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 98
  12. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
11 more
  1. 2022Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decompositioncited by 81
  2. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 74
  3. 2023DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Videocited by 74
  4. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72
  5. 2024TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splattingcited by 60
  6. 2023GaussianHead: High-fidelity Head Avatars with Learnable Gaussian Derivationcited by 46
  7. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesiscited by 41
  8. 2024GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splattingcited by 12
  9. 2025Seeing What Matters: Visual Preference Policy Optimization for Visual Generationcited by 10
  10. 2025Audio Driven Real-Time Facial Animation for Social Telepresencecited by 5
  11. 2025Dual Audio-Centric Modality Coupling for Talking Head Generationcited by 3

Built on by

Not cited yet.