# VIVID-10M: A Dataset and Baseline for Versatile and Interactive Video Local Editing

arXiv: https://arxiv.org/abs/2411.15260 · HTML: https://mlsanity.com/paper/2411.15260/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 38 of its 39 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 38)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion](https://mlsanity.com/paper/2403.06976/) (2024)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing](https://mlsanity.com/paper/2404.09990/) (2024)
- [Negative-prompt Inversion: Fast Image Inversion for Editing with Text-guided Diffusion Models](https://mlsanity.com/paper/2305.16807/) (2023)
- [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](https://mlsanity.com/paper/2312.03594/) (2023)
- [FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing](https://mlsanity.com/paper/2310.05922/) (2023)
- [RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models](https://mlsanity.com/paper/2312.04524/) (2023)

## Built on by (most cited first, 25 of 25)

- [UniVideo: Unified Understanding, Generation, and Editing for Videos](https://mlsanity.com/paper/2510.08377/) (2025)
- [InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Construction](https://mlsanity.com/paper/2503.20287/) (2025)
- [Se~norita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists](https://mlsanity.com/paper/2502.06734/) (2025)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [OpenVE-3M: A Large-Scale High-Quality Dataset for Instruction-Guided Video Editing](https://mlsanity.com/paper/2512.07826/) (2025)
- [MiniMax-Remover: Taming Bad Noise Helps Video Object Removal](https://mlsanity.com/paper/2505.24873/) (2025)
- [VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation](https://mlsanity.com/paper/2503.14350/) (2025)
- [In-Context Learning with Unpaired Clips for Instruction-based Video Editing](https://mlsanity.com/paper/2510.14648/) (2025)
- [EasyV2V: A High-quality Instruction-based Video Editing Framework](https://mlsanity.com/paper/2512.16920/) (2025)
- [MoCha:End-to-End Video Character Replacement without Structural Guidance](https://mlsanity.com/paper/2601.08587/) (2026)
- [Omni-3DEdit: Generalized Versatile 3D Editing in One-Pass](https://mlsanity.com/paper/2603.17841/) (2026)
- [LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localization](https://mlsanity.com/paper/2512.02933/) (2025)
- [FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editing](https://mlsanity.com/paper/2601.01720/) (2026)
- [From Ideal to Real: Stable Video Object Removal under Imperfect Conditions](https://mlsanity.com/paper/2603.09283/) (2026)
- [NOVA: Sparse Control, Dense Synthesis for Pair-Free Video Editing](https://mlsanity.com/paper/2603.02802/) (2026)
- [SteerVTE: Seamless Video Text Editing with Style and Glyph Control](https://mlsanity.com/paper/2606.23254/) (2026)
- [UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolution](https://mlsanity.com/paper/2510.08143/) (2025)
- [Decoupling Complexity from Scale in Latent Diffusion Model](https://mlsanity.com/paper/2511.16117/) (2025)
- [Refaccade: Editing Object with Given Reference Texture](https://mlsanity.com/paper/2512.04534/) (2025)
- [PLACID: Identity-Preserving Multi-Object Compositing via Video Diffusion with Synthetic Trajectories](https://mlsanity.com/paper/2602.00267/) (2026)
- [Vera: A Layered Diffusion Model for Content-Preserving Video Editing](https://mlsanity.com/paper/2606.23610/) (2026)
- [RefVideo-6M: A Reliable Reference-Based Dataset for Instructional Video Editing](https://mlsanity.com/paper/2608.26101/) (2026)
- [From Video Clips to Creation Trajectory: Sora100K for AI-Native Video Creation](https://mlsanity.com/paper/2610.11770/) (2026)
- [VINCIE-NExT: Unlocking Video Editing from Images via In-Context Modeling](https://mlsanity.com/paper/2610.12104/) (2026)
- [QK-Edit: Revisiting Attention-based Injection in MM-DiT for Image and Video Editing](https://mlsanity.com/paper/2610.17829/) (2026)

## Abstract (arXiv)

Diffusion-based image editing models have made remarkable progress in recent years. However, achieving high-quality video editing remains a significant challenge. One major hurdle is the absence of open-source, large-scale video editing datasets based on real-world data, as constructing such datasets is both time-consuming and costly. Moreover, video data requires a significantly larger number of tokens for representation, which substantially increases the training costs for video editing models. Lastly, current video editing models offer limited interactivity, often making it difficult for users to express their editing requirements effectively in a single attempt. To address these challenges, this paper introduces a dataset VIVID-10M and a baseline model VIVID. VIVID-10M is the first large-scale hybrid image-video local editing dataset aimed at reducing data construction and model training costs, which comprises 9.7M samples that encompass a wide range of video editing tasks. VIVID is a Versatile and Interactive VIdeo local eDiting model trained on VIVID-10M, which supports entity addition, modification, and deletion. At its core, a keyframe-guided interactive video editing mechanism is proposed, enabling users to iteratively edit keyframes and propagate it to other frames, thereby reducing latency in achieving desired outcomes. Extensive experimental evaluations show that our approach achieves state-of-the-art performance in video local editing, surpassing baseline methods in both automated metrics and user studies. The VIVID-10M dataset are open-sourced at https://kwaivgi.github.io/VIVID/.