mlsanityData to 7 Oct 2026

Video Editing via Factorized Diffusion Distillation

arXiv 2403.09334 ↗First posted 14 Mar 2024cs.CVcited by 40

On the map: Diffusion models → Video generation

Abstract (arXiv)

We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data. To develop EVE we separately train an image editing adapter and a video generation adapter, and attach both to the same text-to-image model. Then, to align the adapters towards video editing we introduce a new unsupervised distillation procedure, Factorized Diffusion Distillation. This procedure distills knowledge from one or more teachers simultaneously, without any supervised data. We utilize this procedure to teach EVE to edit videos by jointly distilling knowledge to (i) precisely edit each individual frame from the image editing adapter, and (ii) ensure temporal consistency among the edited frames using the video generation adapter. Finally, to demonstrate the potential of our approach in unlocking other capabilities, we align additional combinations of adapters

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 38 of its 42 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

38 of 42 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Lumiere: A Space-Time Diffusion Model for Video Generation23 Jan
  2. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis29 Dec
  3. 2023Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis20 Dec
  4. 2023MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers19 Dec
  5. 2023VidToMe: Video Token Merging for Zero-Shot Video Editing17 Dec
  6. 2023Neural Video Fields Editing12 Dec
  7. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models7 Dec
  8. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation28 Nov
  9. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  10. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  11. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  12. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
2 more
  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  12. 2023Adversarial Diffusion Distillationcited by 741
12 more
  1. 2022On Distillation of Guided Diffusion Modelscited by 721
  2. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  3. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  4. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  5. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  6. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  7. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  8. 2017Geometric GANcited by 232
  9. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  10. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 79
  11. 2023CVPR 2023 Text Guided Video Editing Competitioncited by 54
  12. 2023Motion-Conditioned Image Animation for Video Editingcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 17; 2026: 14.

40 papers.

Sort:

By area:

  1. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  2. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  3. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  4. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 53
  5. 2024Still-Moving: Customized Video Generation without Customized Video Datacited by 49
  6. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  7. 2024Looking Backward: Streaming Video-to-Video Translation with Feature Bankscited by 38
  8. 2024Generative Video Propagationcited by 36
  9. 2026OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoningcited by 25
  10. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 21
  11. 2025FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Modelscited by 15
  12. 2025SketchVideo: Sketch-based Video Generation and Editingcited by 15
28 more
  1. 2025On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devicescited by 14
  2. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 13
  3. 2025EasyV2V: A High-quality Instruction-based Video Editing Frameworkcited by 11
  4. 2025RelightMaster: Precise Video Relighting with Multi-plane Light Imagescited by 8
  5. 2025InstructVEdit: A Holistic Approach for Instructional Video Editingcited by 7
  6. 2025EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editingcited by 7
  7. 2026PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motioncited by 7
  8. 2026Evolution of Video Generative Foundationscited by 6
  9. 2024DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Modelscited by 5
  10. 2025VideoCoF: Unified Video Editing with Temporal Reasonercited by 5
  11. 2024Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editingcited by 3
  12. 2026Versatile Editing of Video Content, Actions, and Dynamics without Trainingcited by 3
  13. 2024Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuningcited by 2
  14. 2025FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editingcited by 2
  15. 2026EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformerscited by 2
  16. 2025Edit-Your-Interest: Efficient Video Editing via Feature Most-Similar Propagationcited by 1
  17. 2026RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editingcited by 1
  18. 2026PropFly: Learning to Propagate via On-the-Fly Supervision from Pre-trained Video Diffusion Modelscited by 1
  19. 2026LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editingcited by 1
  20. 2024PRIMEdit: Probability Redistribution for Instance-aware Multi-object Video Editing with Benchmark Datasetcited by 0
  21. 2025On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devicescited by 0
  22. 2025Enhancing Video Inpainting with Aligned Frame Interval Guidancecited by 0
  23. 2026Beyond Rigid: Benchmarking Non-Rigid Video Editingcited by 0
  24. 2026TDMM-LM: Bridging Facial Understanding and Animation via Language Modelscited by 0
  25. 2026RealMaster: Lifting Rendered Scenes into Photorealistic Videocited by 0
  26. 2026CogOmniControl: Reasoning-Driven Controllable Video Generation via Creative Intent Cognitioncited by 0
  27. 2026EgoPlay: Event-Triggered Video Editing for Egocentric Streamscited by 0
  28. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Temporal Consistency in Diffusion-Based Video Editing with Adaptor Modules: A Theoretical Framework
  2. 2025FADE: Frequency-Aware Diffusion Model Factorization for Video Editing
  3. 2023EVE: Efficient zero-shot text-based Video Editing with Depth Map Guidance and Temporal Consistency Constraints
  4. 2025FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors
  5. 2026Instruction-Based Video Editing by Repurposing an Image Editing Model
  6. 2025DAPE: Dual-Stage Parameter-Efficient Fine-Tuning for Consistent Video Editing with Diffusion Models
  7. 2024DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editing
  8. 2024EffiVED:Efficient Video Editing via Text-instruction Diffusion Models