mlsanityData to 14 Oct 2026

AI killed the video star. Audio-driven diffusion model for expressive talking head generation

arXiv 2511.22488 ↗First posted 27 Nov 2025cs.CVcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose Dimitra++, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we propose a conditional Motion Diffusion Transformer (cMDT) to model facial motion sequences, employing a 3D representation. The cMDT is conditioned on two inputs: a reference facial image, which determines appearance, as well as an audio sequence, which drives the motion. Quantitative and qualitative experiments, as well as a user study on two widely employed datasets, i.e., VoxCeleb2 and CelebV-HQ, suggest that Dimitra++ is able to outperform existing approaches in generating realistic talking heads imparting lip motion, facial expression, and head pose.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 64 of its 77 references that are in the map. First posted on arXiv 27 Nov 2025.

Builds on

64 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis13 Mar
  2. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions27 Feb
  3. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder30 Mar
  4. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan
  5. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors7 Dec

Other references · 59

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  9. 2022Human Motion Diffusion Modelcited by 1,080
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  11. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  12. 2020First Order Motion Model for Image Animationcited by 704
38 more
  1. 2022Equivariant Diffusion for Molecule Generation in 3Dcited by 677
  2. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  3. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
  4. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
  5. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  6. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  7. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  8. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  9. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  10. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  11. 2021Motion Representations for Articulated Animationcited by 255
  12. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 250
  13. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 250
  14. 2022Thin-Plate Spline Motion Model for Image Animationcited by 210
  15. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 203
  16. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  17. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 194
  18. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 187
  19. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 181
  20. 2021Audio-Driven Emotional Video Portraitscited by 180
  21. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 178
  22. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 175
  23. 2018Lip Movements Generation at a Glancecited by 155
  24. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 151
  25. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 149
  26. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  27. 2023DCFace: Synthetic Face Generation with Dual Condition Diffusion Modelcited by 116
  28. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  29. 2019Text-based Editing of Talking-head Videocited by 103
  30. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  31. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  32. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  33. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  34. 2022Imitator: Personalized Speech-driven 3D Facial Animationcited by 74
  35. 2024EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatarscited by 66
  36. 2022SPACE: Speech-driven Portrait Animation with Controllable Expressioncited by 49
  37. 2023HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Facescited by 48
  38. 2023DPE: Disentanglement of Pose and Expression for General Video Portrait Editingcited by 46

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation
  2. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models
  3. 2025MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation
  4. 2025StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
  5. 2024MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation
  6. 2024FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model
  7. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion
  8. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis