# GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer

arXiv: https://arxiv.org/abs/2408.01826 · HTML: https://mlsanity.com/paper/2408.01826/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 3 Aug 2024.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis](https://mlsanity.com/paper/2103.11078/) (2021)
- [Capture, Learning, and Synthesis of 3D Speaking Styles](https://mlsanity.com/paper/1905.03079/) (2019)
- [FaceFormer: Speech-Driven 3D Facial Animation with Transformers](https://mlsanity.com/paper/2112.05329/) (2021)
- [MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement](https://mlsanity.com/paper/2104.08223/) (2021)
- [VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time](https://mlsanity.com/paper/2404.10667/) (2024)
- [CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior](https://mlsanity.com/paper/2301.02379/) (2023)
- [EmoTalk: Speech-Driven Emotional Disentanglement for 3D Face Animation](https://mlsanity.com/paper/2303.11089/) (2023)
- [DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation](https://mlsanity.com/paper/2301.03786/) (2023)
- [Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion](https://mlsanity.com/paper/2204.08451/) (2022)
- [IMAGDressing-v1: Customizable Virtual Dressing](https://mlsanity.com/paper/2407.12705/) (2024)
- [Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert](https://mlsanity.com/paper/2303.17480/) (2023)
- [Emotional Speech-Driven Animation with Content-Emotion Disentanglement](https://mlsanity.com/paper/2306.08990/) (2023)
- [Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis](https://mlsanity.com/paper/2207.11770/) (2022)
- [SpiralNet++: A Fast and Highly Efficient Mesh Convolution Operator](https://mlsanity.com/paper/1911.05856/) (2019)
- [FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusion](https://mlsanity.com/paper/2309.11306/) (2023)
- [SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces](https://mlsanity.com/paper/2306.10799/) (2023)
- [SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory](https://mlsanity.com/paper/2211.00924/) (2022)
- [Diffusion Motion: Generate Text-Guided 3D Human Motion by Diffusion Model](https://mlsanity.com/paper/2210.12315/) (2022)
- [Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers](https://mlsanity.com/paper/2212.04970/) (2022)
- [DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generation](https://mlsanity.com/paper/2312.13578/) (2023)
- [Speech-Driven 3D Face Animation with Composite and Regional Facial Movements](https://mlsanity.com/paper/2308.05428/) (2023)
- [Talking Head from Speech Audio using a Pre-trained Image Generator](https://mlsanity.com/paper/2209.04252/) (2022)
- [Mimic: Speaking Style Disentanglement for Speech-Driven 3D Facial Animation](https://mlsanity.com/paper/2312.10877/) (2023)
- [Learn2Talk: 3D Talking Face Learns from 2D Talking Face](https://mlsanity.com/paper/2404.12888/) (2024)

## Built on by (most cited first, 8 of 8)

- [PESTalk: Speech-Driven 3D Facial Animation with Personalized Emotional Styles](https://mlsanity.com/paper/2512.05121/) (2025)
- [Diverse Code Query Learning for Speech-Driven Facial Animation](https://mlsanity.com/paper/2409.19143/) (2024)
- [3DFacePolicy: Audio-Driven 3D Facial Animation Based on Action Control](https://mlsanity.com/paper/2409.10848/) (2024)
- [Follow Your Motion: A Generic Temporal Consistency Portrait Editing Framework with Trajectory Guidance](https://mlsanity.com/paper/2503.22225/) (2025)
- [EmbodiedHead: Real-Time Listening and Speaking Avatar for Conversational Agents](https://mlsanity.com/paper/2604.17211/) (2026)
- [3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars](https://mlsanity.com/paper/2602.10516/) (2026)
- [ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction](https://mlsanity.com/paper/2609.05506/) (2026)
- [KoUniTalk: A Lightweight Articulation-Centered Korean-English 3D Talking Face Benchmark](https://mlsanity.com/paper/2609.19840/) (2026)

## Abstract (arXiv)

Speech-driven talking head generation is a critical yet challenging task with applications in augmented reality and virtual human modeling. While recent approaches using autoregressive and diffusion-based models have achieved notable progress, they often suffer from modality inconsistencies, particularly misalignment between audio and mesh, leading to reduced motion diversity and lip-sync accuracy. To address this, we propose GLDiTalker, a novel speech-driven 3D facial animation model based on a Graph Latent Diffusion Transformer. GLDiTalker resolves modality misalignment by diffusing signals within a quantized spatiotemporal latent space. It employs a two-stage training pipeline: the Graph-Enhanced Quantized Space Learning Stage ensures lip-sync accuracy, while the Space-Time Powered Latent Diffusion Stage enhances motion diversity. Together, these stages enable GLDiTalker to generate realistic, temporally stable 3D facial animations. Extensive evaluations on standard benchmarks demonstrate that GLDiTalker outperforms existing methods, achieving superior results in both lip-sync accuracy and motion diversity.