mlsanityData to 14 Oct 2026

DiT-Head: High-Resolution Talking Head Synthesis using Diffusion Transformers

arXiv 2312.06400 ↗First posted 11 Dec 2023cs.AI cs.CV cs.LGcited by 4

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose a novel talking head synthesis pipeline called "DiT-Head", which is based on diffusion transformers and uses audio as a condition to drive the denoising process of a diffusion model. Our method is scalable and can generalise to multiple identities while producing high-quality results. We train and evaluate our proposed approach and compare it against existing methods of talking head synthesis. We show that our model can compete with these methods in terms of visual quality and lip-sync accuracy. Our results highlight the potential of our proposed approach to be used for a wide range of applications, including virtual assistants, entertainment, and education. For a video demonstration of the results and our user study, please refer to our supplementary material.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 34 of its 48 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

34 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  2. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Model10 Jan

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2017Neural Discrete Representation Learningcited by 4,712
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
20 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  3. 2014Conditional Generative Adversarial Netscited by 2,162
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  5. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  6. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
  7. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  8. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  9. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 312
  10. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  11. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  12. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  13. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  14. 2023Video Probabilistic Diffusion Models in Projected Latent Spacecited by 171
  15. 2017S3FD: Single Shot Scale-invariant Face Detectorcited by 142
  16. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  17. 2022Latent Diffusion for Language Generationcited by 115
  18. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  19. 2020HeadGAN: One-shot Neural Head Synthesis and Editingcited by 105
  20. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 67
  2. 2025Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functionscited by 6
  3. 2026UIKA: Fast Universal Head Avatar from Pose-Free Imagescited by 5
  4. 2026Giving Faces Their Feelings Back: Explicit Emotion Control for Feedforward Single-Image 3D Head Avatarscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
  2. 2026TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation
  3. 2025AI killed the video star. Audio-driven diffusion model for expressive talking head generation
  4. 2025Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation
  5. 2024Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis
  6. 2024MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generation
  7. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
  8. 2025MoDiT: Learning Highly Consistent 3D Motion Coefficients with Diffusion Transformer for Talking Head Generation