mlsanityData to 7 Oct 2026

Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset

arXiv 2510.15742 ↗First posted 17 Oct 2025cs.CVcited by 76

Not on the map. Why?

Abstract (arXiv)

Instruction-based video editing promises to democratize content creation, yet its progress is severely hampered by the scarcity of large-scale, high-quality training data. We introduce Ditto, a holistic framework designed to tackle this fundamental challenge. At its heart, Ditto features a novel data generation pipeline that fuses the creative diversity of a leading image editor with an in-context video generator, overcoming the limited scope of existing models. To make this process viable, our framework resolves the prohibitive cost-quality trade-off by employing an efficient, distilled model architecture augmented by a temporal enhancer, which simultaneously reduces computational overhead and improves temporal coherence. Finally, to achieve full scalability, this entire pipeline is driven by an intelligent agent that crafts diverse instructions and rigorously filters the output, ensuring quality control at scale. Using this framework, we invested over 12,000 GPU-days to build Ditto-1M, a new dataset of one million high-fidelity video editing examples. We trained our model, Editto, on Ditto-1M with a curriculum learning strategy. The results demonstrate superior instruction-following ability and establish a new state-of-the-art in instruction-based video editing.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Builds on 45 of its 56 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

45 of 56 references.

References · 45

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2022Flow Matching for Generative Modelingcited by 5,426
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  11. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  12. 2025Qwen-Image Technical Reportcited by 973
33 more
  1. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  3. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 605
  4. 2025VACE: All-in-One Video Creation and Editingcited by 467
  5. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  6. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  7. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  8. 2023Improved Techniques for Training Consistency Modelscited by 430
  9. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  10. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 334
  11. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  12. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  13. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  14. 2025Video Depth Anything: Consistent Depth Estimation for Super-Long Videoscited by 202
  15. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  16. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  17. 2025Diffusion Adversarial Post-Training for One-Step Video Generationcited by 142
  18. 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
  19. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  20. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  21. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 79
  22. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  23. 2025Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialistscited by 61
  24. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 53
  25. 2025VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editingcited by 50
  26. 2023InstructVid2Vid: Controllable Video Editing with Natural Language Instructionscited by 44
  27. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  28. 2024VideoDirector: Precise Video Editing via Text-to-Video Modelscited by 36
  29. 2024EffiVED:Efficient Video Editing via Text-instruction Diffusion Modelscited by 21
  30. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  31. 2025InstructVEdit: A Holistic Approach for Instructional Video Editingcited by 7
  32. 2025FADE: Frequency-Aware Diffusion Model Factorization for Video Editingcited by 7
  33. 2025REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 64.

76 papers.

Sort:

By area:

  1. 2026Advancing Open-source World Modelscited by 144
  2. 2025OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editingcited by 50
  3. 2026Kiwi-Edit: Versatile Video Editing via Instruction and Reference Guidancecited by 34
  4. 2026VINO: A Unified Visual Generator with Interleaved OmniModal Contextcited by 26
  5. 2026OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoningcited by 25
  6. 2025Region-Constraint In-Context Generation for Instructional Video Editingcited by 22
  7. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimizationcited by 16
  8. 2026LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editingcited by 15
  9. 2025EasyV2V: A High-quality Instruction-based Video Editing Frameworkcited by 11
  10. 2026Bernini: Latent Semantic Planning for Video Diffusioncited by 11
  11. 2026SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformercited by 10
  12. 2026SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editingcited by 8
38 more
  1. 2026Aurora: Unified Video Editing with a Tool-Using Agentcited by 7
  2. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 6
  3. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  4. 2026SpongeBob: Sync-Aware Harmonious Audio-Visual Generative Editingcited by 6
  5. 2026LoomVideo: Unifying Multimodal Inputs into Video Generation and Editingcited by 6
  6. 2026TeleStyle: Content-Preserving Style Transfer in Images and Videoscited by 5
  7. 2026Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editingcited by 5
  8. 2025Exploring MLLM-Diffusion Information Transfer with MetaCanvascited by 4
  9. 2026InstructAV2AV: Instruction-Guided Audio-Video Joint Editingcited by 4
  10. 2026WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memorycited by 4
  11. 2026JoyAI-Video-Edit: Real-Time Open-Ended Video Editing with Autoregressive Diffusioncited by 4
  12. 2026Mamoda2.5: Enhancing Unified Multimodal Model with DiT-MoEcited by 3
  13. 2026LIVEditor-14B: Lightning Unified Video Editing via In-Context Sparse Attentioncited by 3
  14. 2026CoT-Edit: Let CoT Guide Instruction Video Editingcited by 3
  15. 2025Vision Bridge Transformer at Scalecited by 2
  16. 2025LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localizationcited by 2
  17. 2025ViDiC: Video Difference Captioningcited by 2
  18. 2026Memory-V2V: Memory-Augmented Video-to-Video Diffusion for Consistent Multi-Turn Editingcited by 2
  19. 2026TIDE: Task-Isolated Diffusion for Unified Video Editing and Generationcited by 2
  20. 2026CoVEBench: Can Video Editing Models Handle Complex Instructions?cited by 2
  21. 2026Instruction-Based Video Editing by Repurposing an Image Editing Modelcited by 2
  22. 2026GRNEdit: Efficient General Video Editing from a New Binary-Evidence Perspective in Generative Refinement Networkscited by 2
  23. 2026CoinVE-200K: A Large-Scale High-Quality Dataset for Compositional Instruction-Guided Video Editingcited by 2
  24. 2025MagicWand: A Universal Agent for Generation and Evaluation Aligned with User Preferencecited by 1
  25. 2025Refaccade: Editing Object with Given Reference Texturecited by 1
  26. 2026Learning Stochastic Bridges for Video Object Removal via Video-to-Video Translationcited by 1
  27. 2026Advances in GRPO for Generation Models: A Surveycited by 1
  28. 2026ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformercited by 1
  29. 2026InsEdit: Towards Instruction-based Visual Editing via Data-Efficient Video Diffusion Models Adaptationcited by 1
  30. 2026LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editingcited by 1
  31. 2026Sparkle: Realizing Lively Instruction-Guided Video Background Replacement via Decoupled Guidancecited by 1
  32. 2026JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curationcited by 1
  33. 2026DomainShuttle: Freeform Open Domain Subject-driven Text-to-video Generationcited by 1
  34. 2026OmniEdit-Bench: A Comprehensive Benchmark for Instruction-based Video Editingcited by 1
  35. 2026MSEditor: Toward Consistent Multi-Shot Video Editingcited by 1
  36. 2026Precise Editing and Flexible Referencing for Interactable Worldscited by 1
  37. 2025JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editingcited by 0
  38. 2026RealMaster: Lifting Rendered Scenes into Photorealistic Videocited by 0

50 most cited of 76 All 76 (JSON)