mlsanityData to 7 Oct 2026

DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation

arXiv 2301.03786 ↗First posted 10 Jan 2023cs.CVcited by 136

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Talking head synthesis is a promising approach for the video production industry. Recently, a lot of effort has been devoted in this research area to improve the generation quality or enhance the model generalization. However, there are few works able to address both issues simultaneously, which is essential for practical applications. To this end, in this paper, we turn attention to the emerging powerful Latent Diffusion Models, and model the Talking head generation as an audio-driven temporally coherent denoising process (DiffTalk). More specifically, instead of employing audio signals as the single driving factor, we investigate the control mechanism of the talking face, and incorporate reference face images and landmarks as conditions for personality-aware generalized synthesis. In this way, the proposed DiffTalk is capable of producing high-quality talking head videos in synchronization with the source audio, and more importantly, it can be naturally generalized across different identities without any further fine-tuning. Additionally, our DiffTalk can be gracefully tailored for higher-resolution synthesis with negligible extra computational cost. Extensive experiments show that the proposed DiffTalk efficiently synthesizes high-fidelity audio-driven talking head videos for generalized novel identities. For more video results, please refer to urlhttps://sstzal.github.io/DiffTalk/.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 43 of its 58 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

43 of 58 references.

References · 43

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  12. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
31 more
  1. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  2. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  3. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  4. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  5. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  6. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  7. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  8. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  9. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  10. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  11. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  12. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  13. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  14. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  15. 2020Real-Time Intermediate Flow Estimation for Video Frame Interpolationcited by 211
  16. 2021Audio-Driven Emotional Video Portraitscited by 180
  17. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  18. 2022Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generationcited by 119
  19. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  20. 2022DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latentscited by 103
  21. 2022Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesiscited by 97
  22. 2020Towards Automatic Face-to-Face Translationcited by 95
  23. 2020Everybody's Talkin': Let Me Talk as You Wantcited by 83
  24. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  25. 2022Text-Guided Synthesis of Artistic Images with Retrieval-Augmented Diffusion Modelscited by 63
  26. 2022DiffuStereo: High Quality Human Reconstruction via Diffusion-based Stereo Using Sparse Camerascited by 43
  27. 2022A generic diffusion-based approach for 3D human pose prediction in the wildcited by 33
  28. 2019FLNet: Landmark Driven Fetching and Learning Network for Faithful Talking Facial Animation Synthesiscited by 28
  29. 2022A Diffusion Model Predicts 3D Shapes from 2D Microscopy Imagescited by 24
  30. 2022Face Super-Resolution Using Stochastic Differential Equationscited by 10
  31. 2022Intelligent Painter: Picture Composition With Resampling Diffusion Modelcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 50; 2025: 48; 2026: 20.

136 papers.

Sort:

By area:

  1. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  2. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 202
  3. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  4. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  5. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  6. 2023AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Datasetcited by 79
  7. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  8. 2025OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animationcited by 77
  9. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  10. 2023Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesiscited by 72
  11. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70
  12. 2023FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Modelscited by 55
38 more
  1. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronizationcited by 49
  2. 2023GAIA: Zero-shot Talking Avatar Generationcited by 43
  3. 2025MoCha: Towards Movie-Grade Talking Character Synthesiscited by 31
  4. 2023PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Modelscited by 30
  5. 2024Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generationcited by 29
  6. 2024MuseTalk: Real-Time High-Fidelity Video Dubbing via Spatio-Temporal Samplingcited by 27
  7. 2025Human Motion Video Generation: A Surveycited by 25
  8. 2023DREAM-Talk: Diffusion-based Realistic Emotional Audio-driven Method for Single Image Talking Face Generationcited by 24
  9. 2024Self-Guidance: Boosting Flow and Diffusion Generation on Their Owncited by 22
  10. 2023AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesiscited by 16
  11. 2024ReSyncer: Rewiring Style-based Generator for Unified Audio-Visually Synced Facial Performercited by 16
  12. 2024EmotiveTalk: Expressive Talking Head Generation through Audio Information Decoupling and Emotional Video Diffusioncited by 16
  13. 2025Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generationcited by 16
  14. 2025Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimizationcited by 16
  15. 2023Interactive Conversational Head Generationcited by 14
  16. 2024DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactmentcited by 14
  17. 2024RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Networkcited by 14
  18. 2025FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Casescited by 14
  19. 2023Audio-driven Talking Face Generation with Stabilized Synchronization Losscited by 13
  20. 2023A Generalist FaceX via Learning Unified Facial Representationcited by 13
  21. 2024DiffSal: Joint Audio and Video Learning for Diffusion Saliency Predictioncited by 13
  22. 2025MoEE: Mixture of Emotion Experts for Audio-Driven Portrait Animationcited by 12
  23. 2025SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusioncited by 12
  24. 2024EmoTalker: Emotionally Editable Talking Face Generation via Diffusion Modelcited by 11
  25. 2025Disentangle Identity, Cooperate Emotion: Correlation-Aware Emotional Talking Portrait Generationcited by 11
  26. 2025KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolutioncited by 11
  27. 2023DiffDub: Person-generic Visual Dubbing Using Inpainting Renderer with Diffusion Auto-encodercited by 10
  28. 2024MoDiTalker: Motion-Disentangled Diffusion Model for High-Fidelity Talking Head Generationcited by 10
  29. 2023SingingHead: A Large-scale 4D Dataset for Singing Head Animationcited by 9
  30. 2024UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Modelscited by 9
  31. 2025AV-Flow: Transforming Text to Audio-Visual Human-like Interactionscited by 9
  32. 2025SyncTalk++: High-Fidelity and Efficient Synchronized Talking Heads Synthesis Using Gaussian Splattingcited by 9
  33. 2024GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformercited by 8
  34. 2024DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generationcited by 8
  35. 2025Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusioncited by 8
  36. 2024DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generationcited by 7
  37. 2024DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesiscited by 7
  38. 2025Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generationcited by 7

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DiT-Head: High-Resolution Talking Head Synthesis using Diffusion Transformers
  2. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Models
  3. 2023DiffTalker: Co-driven audio-image diffusion for talking faces via intermediate landmarks
  4. 2024DreamHead: Learning Spatial-Temporal Correspondence via Hierarchical Diffusion for Audio-driven Talking Head Synthesis
  5. 2024GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expression
  6. 2025DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis
  7. 2025ConsistTalk: Intensity Controllable Temporally Consistent Talking Head Generation with Diffusion Noise Search
  8. 2023From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applications