# Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset

arXiv: https://arxiv.org/abs/2510.15742 · HTML: https://mlsanity.com/paper/2510.15742/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 56 references that are in the map.
- First posted on arXiv 17 Oct 2025.

## Builds on (most cited first, 30 of 45)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Improved Techniques for Training Consistency Models](https://mlsanity.com/paper/2310.14189/) (2023)
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](https://mlsanity.com/paper/2412.07772/) (2024)
- [OminiControl: Minimal and Universal Control for Diffusion Transformer](https://mlsanity.com/paper/2411.15098/) (2024)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)
- [ReCamMaster: Camera-Controlled Generative Rendering from A Single Video](https://mlsanity.com/paper/2503.11647/) (2025)
- [Video Depth Anything: Consistent Depth Estimation for Super-Long Videos](https://mlsanity.com/paper/2501.12375/) (2025)
- [StableVideo: Text-driven Consistency-aware Diffusion Video Editing](https://mlsanity.com/paper/2308.09592/) (2023)
- [Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model](https://mlsanity.com/paper/2502.10248/) (2025)
- [Diffusion Adversarial Post-Training for One-Step Video Generation](https://mlsanity.com/paper/2501.08316/) (2025)
- [CoDeF: Content Deformation Fields for Temporally Consistent Video Processing](https://mlsanity.com/paper/2308.07926/) (2023)

## Built on by (most cited first, 30 of 79)

- [Advancing Open-source World Models](https://mlsanity.com/paper/2601.20540/) (2026)
- [OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing](https://mlsanity.com/paper/2512.07826/) (2025)
- [Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidance](https://mlsanity.com/paper/2603.02175/) (2026)
- [OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning](https://mlsanity.com/paper/2603.24458/) (2026)
- [VINO: A Unified Visual Generator with Interleaved OmniModal Context](https://mlsanity.com/paper/2601.02358/) (2026)
- [Region-Constraint In-Context Generation for Instructional Video Editing](https://mlsanity.com/paper/2512.17650/) (2025)
- [VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization](https://mlsanity.com/paper/2512.16906/) (2025)
- [LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing](https://mlsanity.com/paper/2606.26740/) (2026)
- [EasyV2V: A High-quality Instruction-based Video Editing Framework](https://mlsanity.com/paper/2512.16920/) (2025)
- [Bernini: Latent Semantic Planning for Video Diffusion](https://mlsanity.com/paper/2605.22344/) (2026)
- [SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformer](https://mlsanity.com/paper/2605.30409/) (2026)
- [Aurora: Unified Video Editing with a Tool-Using Agent](https://mlsanity.com/paper/2605.18748/) (2026)
- [SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing](https://mlsanity.com/paper/2603.19228/) (2026)
- [SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editing](https://mlsanity.com/paper/2605.25193/) (2026)
- [Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learning](https://mlsanity.com/paper/2512.09924/) (2025)
- [LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation](https://mlsanity.com/paper/2604.11789/) (2026)
- [LoomVideo: Unifying Multimodal Inputs into Video Generation and Editing](https://mlsanity.com/paper/2606.06042/) (2026)
- [Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing](https://mlsanity.com/paper/2606.30599/) (2026)
- [JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusion](https://mlsanity.com/paper/2608.03974/) (2026)
- [LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localization](https://mlsanity.com/paper/2512.02933/) (2025)
- [TeleStyle: Content-Preserving Style Transfer in Images and Videos](https://mlsanity.com/paper/2601.20175/) (2026)
- [InstructAV2AV: Instruction-Guided Audio-Video Joint Editing](https://mlsanity.com/paper/2605.18467/) (2026)
- [WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory](https://mlsanity.com/paper/2607.02517/) (2026)
- [Exploring MLLM-Diffusion Information Transfer with MetaCanvas](https://mlsanity.com/paper/2512.11464/) (2025)
- [AutoWeather4D: Autonomous Driving Video Weather Conversion via G-Buffer Dual-Pass Editing](https://mlsanity.com/paper/2603.26546/) (2026)
- [LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attention](https://mlsanity.com/paper/2605.04569/) (2026)
- [Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editing](https://mlsanity.com/paper/2601.16296/) (2026)
- [Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoE](https://mlsanity.com/paper/2605.02641/) (2026)
- [CoVEBench: Can Video Editing Models Handle Complex Instructions?](https://mlsanity.com/paper/2606.08415/) (2026)
- [CoT-Edit: Let CoT Guide Instruction Video Editing](https://mlsanity.com/paper/2608.01113/) (2026)

## Abstract (arXiv)

Instruction-based video editing promises to democratize content creation, yet its progress is severely hampered by the scarcity of large-scale, high-quality training data. We introduce Ditto, a holistic framework designed to tackle this fundamental challenge. At its heart, Ditto features a novel data generation pipeline that fuses the creative diversity of a leading image editor with an in-context video generator, overcoming the limited scope of existing models. To make this process viable, our framework resolves the prohibitive cost-quality trade-off by employing an efficient, distilled model architecture augmented by a temporal enhancer, which simultaneously reduces computational overhead and improves temporal coherence. Finally, to achieve full scalability, this entire pipeline is driven by an intelligent agent that crafts diverse instructions and rigorously filters the output, ensuring quality control at scale. Using this framework, we invested over 12,000 GPU-days to build Ditto-1M, a new dataset of one million high-fidelity video editing examples. We trained our model, Editto, on Ditto-1M with a curriculum learning strategy. The results demonstrate superior instruction-following ability and establish a new state-of-the-art in instruction-based video editing.