mlsanityData to 7 Oct 2026

Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation

arXiv 2301.03396 ↗First posted 6 Jan 2023cs.CVcited by 142

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos. Recent developments in diffusion-based generative models allow for more realistic and stable data synthesis and their performance on image and video generation has surpassed that of other generative models. In this work, we present an autoregressive diffusion model that requires only one identity image and audio sequence to generate a video of a realistic talking human head. Our solution is capable of hallucinating head movements, facial expressions, such as blinks, and preserving a given background. We evaluate our model on two different datasets, achieving state-of-the-art results on both of them.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 35 of its 53 references that are in the map. First posted on arXiv 6 Jan 2023.

Builds on

35 of 53 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  12. 2022Video Diffusion Modelscited by 2,280
23 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  5. 2020First Order Motion Model for Image Animationcited by 698
  6. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  7. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  8. 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
  9. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  10. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  11. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  12. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  13. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  14. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  15. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  16. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  17. 2018GANimation: Anatomically-aware Facial Animation from a Single Imagecited by 160
  18. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  19. 2019Realistic Speech-Driven Facial Animation with GANscited by 142
  20. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  21. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  22. 2021LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalizationcited by 67
  23. 2018Generative Adversarial Talking Head: Bringing Portraits to Life with a Weakly Supervised Neural Networkcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 54; 2025: 49; 2026: 13.

142 papers.

Sort:

By area:

  1. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  2. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  3. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  4. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  5. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  6. 2024V-Express: Conditional Dropout for Progressive Training of Portrait Video Generationcited by 85
  7. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  8. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
  9. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72
  10. 2024AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encodingcited by 67
  11. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  12. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
38 more
  1. 2023DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencodercited by 48
  2. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  3. 2023AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensicscited by 46
  4. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  5. 2023DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatarscited by 42
  6. 2024ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preservingcited by 42
  7. 2024FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portraitcited by 40
  8. 2024INFP: Audio-Driven Interactive Head Generation in Dyadic Conversationscited by 40
  9. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attentioncited by 38
  10. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generationcited by 38
  11. 2024Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Stylecited by 33
  12. 2025OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulationcited by 30
  13. 2023FaceXHuBERT: Text-less Speech-driven E(X)pressive 3D Facial Animation Synthesis Using Self-Supervised Speech Representation Learningcited by 29
  14. 2023MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentionscited by 29
  15. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Modelcited by 28
  16. 2024Video Diffusion Models: A Surveycited by 28
  17. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformerscited by 26
  18. 2025Human Motion Video Generation: A Surveycited by 25
  19. 2023DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generationcited by 24
  20. 2023DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generationcited by 20
  21. 2024FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audiocited by 20
  22. 2025InterActHuman: Multi-Concept Human Animation with Layout-Aligned Audio Conditionscited by 20
  23. 2024PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preferencecited by 17
  24. 2024ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performercited by 16
  25. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusioncited by 16
  26. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  27. 2024PersonaTalk: Bring Attention to Your Persona in Visual Dubbingcited by 15
  28. 2024DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactmentcited by 14
  29. 2023Audio-driven Talking Face Generation with Stabilized Synchronization Losscited by 13
  30. 2023DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Modelscited by 13
  31. 2024MegActor-Sigma: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformercited by 13
  32. 2024ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAEcited by 13
  33. 2024Balanced Mixed-Type Tabular Data Synthesis with Diffusion Modelscited by 12
  34. 2024GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splattingcited by 12
  35. 2025MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animationcited by 12
  36. 2025KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolutioncited by 11
  37. 2025KlingAvatar 2.0 Technical Reportcited by 11
  38. 2023DiffDub: Person-generic Visual Dubbing Using Inpainting Renderer with Diffusion Auto-encodercited by 10

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applications
  2. 2025MoDA: Multi-modal Diffusion Architecture for Talking Head Generation
  3. 2022Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors
  4. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
  5. 2025FLAP: Fully-controllable Audio-driven Portrait Video Generation through 3D head conditioned diffusion model
  6. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
  7. 2023Realistic Speech-to-Face Generation with Speech-Conditioned Latent Diffusion Model with Face Prior
  8. 2023Face Animation with an Attribute-Guided Diffusion Model