mlsanityData to 14 Oct 2026

Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation

arXiv 2502.17198 ↗First posted 24 Feb 2025cs.CVcited by 4

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose Dimitra, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we train a conditional Motion Diffusion Transformer (cMDT) by modeling facial motion sequences with 3D representation. We condition the cMDT with only two input signals, an audio-sequence, as well as a reference facial image. By extracting additional features directly from audio, Dimitra is able to increase quality and realism of generated videos. In particular, phoneme sequences contribute to the realism of lip motion, whereas text transcript to facial expression and head pose realism. Quantitative and qualitative experiments on two widely employed datasets, VoxCeleb2 and HDTF, showcase that Dimitra is able to outperform existing approaches for generating realistic talking heads imparting lip motion, facial expression, and head pose.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 35 of its 43 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

35 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models30 Sep
  2. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder30 Mar
  3. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation10 Jan
  4. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan
  5. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors7 Dec

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  6. 2020First Order Motion Model for Image Animationcited by 704
  7. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  8. 2019Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Setcited by 482
  9. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  10. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  11. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  12. 2022Flexible Diffusion Modeling of Long Videoscited by 294
18 more
  1. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  2. 2022Thin-Plate Spline Motion Model for Image Animationcited by 210
  3. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 203
  4. 2021Audio-Driven Emotional Video Portraitscited by 180
  5. 2022Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 178
  6. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 175
  7. 2018Lip Movements Generation at a Glancecited by 155
  8. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 149
  9. 2023DCFace: Synthetic Face Generation with Dual Condition Diffusion Modelcited by 116
  10. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  11. 2019Text-based Editing of Talking-head Videocited by 103
  12. 2024Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesiscited by 97
  13. 2022Progressive Disentangled Representation Learning for Fine-Grained Controllable Talking Head Synthesiscited by 93
  14. 2023StyleTalk: One-shot Talking Head Generation with Controllable Speaking Stylescited by 92
  15. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  16. 2022SPACE: Speech-driven Portrait Animation with Controllable Expressioncited by 49
  17. 2022Shifted Diffusion for Text-to-image Generationcited by 37
  18. 2023AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesiscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025THEval. Evaluation Framework for Talking Head Video Generationcited by 3
  2. 2025Preview WB-DH: Towards Whole Body Digital Human Bench for the Generation of Whole-body Talking Avatar Videoscited by 0
  3. 2026Talking Together: Synthesizing Co-Located 3D Conversations from Audiocited by 0
  4. 2026CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AI killed the video star. Audio-driven diffusion model for expressive talking head generation
  2. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models
  3. 2025MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation
  4. 2024MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation
  5. 2025StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
  6. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusion
  7. 2024DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation
  8. 2024DreamHead: Learning Spatial-Temporal Correspondence via Hierarchical Diffusion for Audio-driven Talking Head Synthesis