# Video Editing via Factorized Diffusion Distillation

arXiv: https://arxiv.org/abs/2403.09334 · HTML: https://mlsanity.com/paper/2403.09334/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 38 of its 42 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 38)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://mlsanity.com/paper/2305.08891/) (2023)
- [Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack](https://mlsanity.com/paper/2309.15807/) (2023)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://mlsanity.com/paper/2306.07954/) (2023)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models](https://mlsanity.com/paper/2303.17599/) (2023)
- [Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer](https://mlsanity.com/paper/2311.17009/) (2023)
- [RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models](https://mlsanity.com/paper/2312.04524/) (2023)

## Built on by (most cited first, 30 of 41)

- [ReVideo: Remake a Video with Motion and Content Control](https://mlsanity.com/paper/2405.13865/) (2024)
- [InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction](https://mlsanity.com/paper/2503.20287/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [Still-Moving: Customized Video Generation without Customized Video Data](https://mlsanity.com/paper/2407.08674/) (2024)
- [Diffusion Model-Based Video Editing: A Survey](https://mlsanity.com/paper/2407.07111/) (2024)
- [Looking Backward: Streaming Video-to-Video Translation with Feature Banks](https://mlsanity.com/paper/2405.15757/) (2024)
- [Generative Video Propagation](https://mlsanity.com/paper/2412.19761/) (2024)
- [OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning](https://mlsanity.com/paper/2603.24458/) (2026)
- [FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generation](https://mlsanity.com/paper/2506.01144/) (2025)
- [FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Models](https://mlsanity.com/paper/2503.13684/) (2025)
- [SketchVideo: Sketch-based Video Generation and Editing](https://mlsanity.com/paper/2503.23284/) (2025)
- [On-device Sora: Enabling Training-Free Diffusion-based Text-to-Video Generation for Mobile Devices](https://mlsanity.com/paper/2502.04363/) (2025)
- [Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising](https://mlsanity.com/paper/2503.20782/) (2025)
- [EasyV2V: A High-quality Instruction-based Video Editing Framework](https://mlsanity.com/paper/2512.16920/) (2025)
- [RelightMaster: Precise Video Relighting with Multi-plane Light Images](https://mlsanity.com/paper/2511.06271/) (2025)
- [EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing](https://mlsanity.com/paper/2512.06065/) (2025)
- [VideoCoF: Unified Video Editing with Temporal Reasoner](https://mlsanity.com/paper/2512.07469/) (2025)
- [PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion](https://mlsanity.com/paper/2606.01399/) (2026)
- [InstructVEdit: A Holistic Approach for Instructional Video Editing](https://mlsanity.com/paper/2503.17641/) (2025)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [DreamSteerer: Enhancing Source Image Conditioned Editability using Personalized Diffusion Models](https://mlsanity.com/paper/2410.11208/) (2024)
- [Versatile Editing of Video Content, Actions, and Dynamics without Training](https://mlsanity.com/paper/2603.17989/) (2026)
- [Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing](https://mlsanity.com/paper/2410.12526/) (2024)
- [Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuning](https://mlsanity.com/paper/2412.02114/) (2024)
- [FluencyVE: Marrying Temporal-Aware Mamba with Bypass Attention for Video Editing](https://mlsanity.com/paper/2512.21015/) (2025)
- [EditYourself: Audio-Driven Generation and Manipulation of Talking Head Videos with Diffusion Transformers](https://mlsanity.com/paper/2601.22127/) (2026)
- [Edit-Your-Interest: Efficient Video Editing via Feature Most-Similar Propagation](https://mlsanity.com/paper/2510.13084/) (2025)
- [RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editing](https://mlsanity.com/paper/2602.06871/) (2026)
- [PropFly: Learning to Propagate via On-the-Fly Supervision from Pre-trained Video Diffusion Models](https://mlsanity.com/paper/2602.20583/) (2026)

## Abstract (arXiv)

We introduce Emu Video Edit (EVE), a model that establishes a new state-of-the art in video editing without relying on any supervised video editing data. To develop EVE we separately train an image editing adapter and a video generation adapter, and attach both to the same text-to-image model. Then, to align the adapters towards video editing we introduce a new unsupervised distillation procedure, Factorized Diffusion Distillation. This procedure distills knowledge from one or more teachers simultaneously, without any supervised data. We utilize this procedure to teach EVE to edit videos by jointly distilling knowledge to (i) precisely edit each individual frame from the image editing adapter, and (ii) ensure temporal consistency among the edited frames using the video generation adapter. Finally, to demonstrate the potential of our approach in unlocking other capabilities, we align additional combinations of adapters