mlsanityData to 7 Oct 2026

EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers

arXiv 2601.22127 ↗First posted 29 Jan 2026cs.CV cs.GR cs.LGcited by 2

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Current generative video models excel at producing novel content from text and image prompts, but leave a critical gap in editing existing pre-recorded videos, where minor alterations to the spoken script require preserving motion, temporal coherence, speaker identity, and accurate lip synchronization. We introduce EditYourself, a DiT-based framework for audio-driven video-to-video (V2V) editing that enables transcript-based modification of talking head videos, including the seamless addition, removal, and retiming of visually spoken content. Building on a general-purpose video diffusion model, EditYourself augments its V2V capabilities with audio conditioning and region-aware, edit-focused training extensions. This enables precise lip synchronization and temporally coherent restructuring of existing performances via spatiotemporal inpainting, including the synthesis of realistic human motion in newly added segments, while maintaining visual fidelity and identity consistency over long durations. This work represents a foundational step toward generative video models as practical tools for professional video post-production.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 93 of its 113 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

93 of 113 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025RAP: Real-time Audio-driven Portrait Animation with Video Diffusion Transformer7 Aug
  2. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers27 May
  3. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation3 Apr
  4. 2025Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion11 Feb
  5. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervision12 Dec
  6. 2024MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation5 Dec
  7. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformer1 Dec
  8. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependency4 Sep
  9. 2024Text-based Talking Video Editing with Cascaded Conditional Diffusion20 Jul

Other references · 84

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  12. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
38 more
  1. 2022Video Diffusion Modelscited by 2,280
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  3. 2017Decoupled Weight Decay Regularizationcited by 1,722
  4. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  5. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  6. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  7. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 669
  8. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  9. 2025VACE: All-in-One Video Creation and Editingcited by 467
  10. 2020Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstructioncited by 465
  11. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  12. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  13. 2019Capture, Learning, and Synthesis of 3D Speaking Stylescited by 257
  14. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 245
  15. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  16. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  17. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  18. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 193
  19. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  20. 2023CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Priorcited by 183
  21. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  22. 2018Lip Movements Generation at a Glancecited by 154
  23. 2023GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesiscited by 131
  24. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  25. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 126
  26. 2019Text-based Editing of Talking-head Videocited by 102
  27. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  28. 2023Elucidating the Exposure Bias in Diffusion Modelscited by 98
  29. 2020Towards Automatic Face-to-Face Translationcited by 95
  30. 2023StyleSync: High-Fidelity Generalized and Personalized Lip Sync in Style-based Generatorcited by 89
  31. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 81
  32. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  33. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Modelscited by 80
  34. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesiscited by 79
  35. 2025FantasyTalking: Realistic Talking Portrait Generation via Coherent Motion Synthesiscited by 77
  36. 2025OmniAvatar: Efficient Audio-Driven Avatar Video Generation with Adaptive Body Animationcited by 77
  37. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70
  38. 2025The Principles of Diffusion Modelscited by 70

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRAcited by 5
  2. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion
  2. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Model
  3. 2025DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis
  4. 2023DiffTalk: Crafting Diffusion Models for Generalized Audio-Driven Portraits Animation
  5. 2025From Inpainting to Editing: Unlocking Robust Mask-Free Visual Dubbing via Generative Bootstrapping
  6. 2024PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation
  7. 2025DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion
  8. 2026Decoupled Self-Forcing Distillation for Streaming Talking Head Generation