# Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing

arXiv: https://arxiv.org/abs/2606.30599 · HTML: https://mlsanity.com/paper/2606.30599/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 38 references that are in the map.
- First posted on arXiv 29 Jun 2026.

## Builds on (most cited first, 30 of 34)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos](https://mlsanity.com/paper/2304.01186/) (2023)
- [Photorealistic Video Generation with Diffusion Models](https://mlsanity.com/paper/2312.06662/) (2023)
- [ReCamMaster: Camera-Controlled Generative Rendering from A Single Video](https://mlsanity.com/paper/2503.11647/) (2025)
- [TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation](https://mlsanity.com/paper/2412.03069/) (2024)
- [Video Generation From Text](https://mlsanity.com/paper/1710.00421/) (2017)
- [AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks](https://mlsanity.com/paper/2403.14468/) (2024)
- [Diffusion Adversarial Post-Training for One-Step Video Generation](https://mlsanity.com/paper/2501.08316/) (2025)
- [Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content](https://mlsanity.com/paper/2410.08260/) (2024)
- [Generating Long Videos of Dynamic Scenes](https://mlsanity.com/paper/2206.03429/) (2022)
- [Consistent Video-to-Video Transfer Using Synthetic Dataset](https://mlsanity.com/paper/2311.00213/) (2023)
- [Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset](https://mlsanity.com/paper/2510.15742/) (2025)
- [Allegro: Open the Black Box of Commercial-Level Video Generation Model](https://mlsanity.com/paper/2410.15458/) (2024)
- [InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction](https://mlsanity.com/paper/2503.20287/) (2025)
- [Follow-Your-Creation: Empowering 4D Creation through Video Inpainting](https://mlsanity.com/paper/2506.04590/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing](https://mlsanity.com/paper/2512.07826/) (2025)

## Built on by (most cited first, 6 of 6)

- [Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention](https://mlsanity.com/paper/2608.15522/) (2026)
- [RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing](https://mlsanity.com/paper/2608.26101/) (2026)
- [Scaling Versatile 3D Assets Editing with a Million-Scale Dataset](https://mlsanity.com/paper/2609.34271/) (2026)
- [AVIO: Learning to Add and Remove Sounding Objects in Audiovisual Scenes](https://mlsanity.com/paper/2609.36503/) (2026)
- [ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing](https://mlsanity.com/paper/2610.08779/) (2026)
- [Towards Unified Evaluation of Prompt Enhancers for Video Generation](https://mlsanity.com/paper/2610.11736/) (2026)

## Abstract (arXiv)

Existing instruction-based video editing datasets commonly focus on single-task appearance editing, failing to meet the complex creative demands of real-world scenarios. To bridge this gap, we present Goku, a large-scale dataset featuring 2 million high-quality, instruction-aligned video editing pairs, which is the first to extend task boundaries from basic appearance editing to multi-task and structural manipulations(e.g., precise control of subject movement). To tackle the data synthesis challenges inherent in these complex tasks, we design an efficient data synthesis pipeline that decomposes complex edits into controllable sub-problems and introduce a progressive filtering system for data reliability throughout the whole process. Furthermore, we explore the optimal network structures on Goku, and propose Goku-Edit. To deeply comprehend complex editing instructions, Goku-Edit leverages an MLLM as its text encoder and adopts a decoupled dual-branch design: a dedicated mask branch handles structural control, freeing the main branch for appearance rendering. A comprehensive video editing benchmark, Goku-Bench, is also proposed with 1,000 human-verified test cases and 7 novel editing-specific metrics. Evaluated on Goku-Bench, Goku-Edit obtains up to +8% improvement on other open-source models in terms of instruction following.