mlsanityData to 7 Oct 2026

Speech Driven Video Editing via an Audio-Conditioned Diffusion Model

arXiv 2301.04474 ↗First posted 10 Jan 2023cs.CV cs.LG cs.SDcited by 28

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Taking inspiration from recent developments in visual generative tasks using diffusion models, we propose a method for end-to-end speech-driven video editing using a denoising diffusion model. Given a video of a talking person, and a separate auditory speech recording, the lip and jaw motions are re-synchronized without relying on intermediate structural representations such as facial landmarks or a 3D face model. We show this is possible by conditioning a denoising diffusion model on audio mel spectral features to generate synchronised facial motion. Proof of concept results are demonstrated on both single-speaker and multi-speaker video editing, providing a baseline model on the CREMA-D audiovisual data set. To the best of our knowledge, this is the first work to demonstrate and validate the feasibility of applying end-to-end denoising diffusion models to the task of audio-driven video editing.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 80 of its 106 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

80 of 106 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation6 Jan
  2. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  3. 2022ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech13 Jul
  4. 2022Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models5 Jun
  5. 2022Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data30 May
  6. 2021EdiTTS: Score-based Editing for Controllable Text-to-Speech6 Oct
  7. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech13 May
  8. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesis21 Sep

Other references · 72

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022Generative Adversarial Networkscited by 5,603
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
38 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2022Video Diffusion Modelscited by 2,280
  5. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  6. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  7. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  10. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  11. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  12. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  13. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  14. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  15. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  16. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  17. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 572
  18. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  19. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  20. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  21. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447
  22. 2020MakeItTalk: Speaker-Aware Talking-Head Animationcited by 310
  23. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  24. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  25. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  26. 2019Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Losscited by 252
  27. 2019Neural Voice Puppetry: Audio-driven Facial Reenactmentcited by 249
  28. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  29. 2020DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detectioncited by 231
  30. 2016Lip Reading Sentences in the Wildcited by 226
  31. 2018Talking Face Generation by Adversarially Disentangled Audio-Visual Representationcited by 212
  32. 2021Conditional Image Generation with Score-Based Diffusion Modelscited by 195
  33. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  34. 2021Audio-Driven Emotional Video Portraitscited by 180
  35. 2018Lip Movements Generation at a Glancecited by 154
  36. 2022EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Modelcited by 151
  37. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 148
  38. 2019Realistic Speech-Driven Facial Animation with GANscited by 142

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 10; 2025: 7; 2026: 3.

28 papers.

Sort:

By area:

  1. 2023A Survey on Video Diffusion Modelscited by 203
  2. 2023FaceDiffuser: Speech-Driven 3D Facial Animation Synthesis Using Diffusioncited by 77
  3. 2024AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encodingcited by 67
  4. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 53
  5. 2025Human Motion Video Generation: A Surveycited by 25
  6. 2023DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Modelscited by 13
  7. 2026Hallo-Live: Real-Time Streaming Joint Audio-Video Avatar Generation with Asynchronous Dual-Stream and Human-Centric Preference Distillationcited by 13
  8. 2025KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolutioncited by 11
  9. 2023DiffDub: Person-generic Visual Dubbing Using Inpainting Renderer with Diffusion Auto-encodercited by 10
  10. 2023Multimodal-driven Talking Face Generation via a Unified Diffusion-based Generatorcited by 9
  11. 2025Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusioncited by 8
  12. 2024DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generationcited by 7
16 more
  1. 2023HyperLips: Hyper Control Lips with High Resolution Decoder for Talking Face Generationcited by 6
  2. 2025Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functionscited by 6
  3. 2023DiT-Head: High-Resolution Talking Head Synthesis using Diffusion Transformerscited by 4
  4. 2024A Comprehensive Taxonomy and Analysis of Talking Head Synthesis: Techniques for Portrait Generation, Driving Mechanisms, and Editingcited by 4
  5. 2024Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speechcited by 4
  6. 2024Takin-ADA: Emotion Controllable Audio-Driven Animation with Canonical and Landmark Loss Optimizationcited by 4
  7. 2023Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realismcited by 3
  8. 2024Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generationcited by 2
  9. 2024High-fidelity and Lip-synced Talking Face Synthesis via Landmark-based Diffusion Modelcited by 1
  10. 2025Removing Averaging: Personalized Lip-Sync Driven Characters Based on Identity Adaptercited by 1
  11. 2025RASA: Replace Anyone, Say Anything -- A Training-Free Framework for Audio-Driven and Universal Portrait Video Editingcited by 1
  12. 2026AUHead: Realistic Emotional Talking Head Generation via Action Units Controlcited by 1
  13. 2026Geometry-guided Emotion Modulation for Controllable and Photorealistic Emotional Talking Face Generationcited by 1
  14. 2024Embedded Representation Learning Network for Animating Styled Video Portraitcited by 0
  15. 2024LatentColorization: Latent Diffusion-Based Speaker Video Colorizationcited by 0
  16. 2025FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20233DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing
  2. 2026EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers
  3. 2023SAiD: Speech-driven Blendshape Facial Animation with Diffusion
  4. 2026TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation
  5. 2025AI killed the video star. Audio-driven diffusion model for expressive talking head generation
  6. 2026JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion
  7. 2025StreamingTalker: Audio-driven 3D Facial Animation with Autoregressive Diffusion Model
  8. 2024DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer