# InstructVEdit: A Holistic Approach for Instructional Video Editing

arXiv: https://arxiv.org/abs/2503.17641 · HTML: https://mlsanity.com/paper/2503.17641/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 43 references that are in the map.
- First posted on arXiv 22 Mar 2025.

## Builds on (most cited first, 30 of 42)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [UltraEdit: Instruction-based Fine-Grained Image Editing at Scale](https://mlsanity.com/paper/2407.05282/) (2024)
- [MOSE: A New Dataset for Video Object Segmentation in Complex Scenes](https://mlsanity.com/paper/2302.01872/) (2023)
- [MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions](https://mlsanity.com/paper/2308.08544/) (2023)
- [HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing](https://mlsanity.com/paper/2404.09990/) (2024)

## Built on by (most cited first, 7 of 7)

- [Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset](https://mlsanity.com/paper/2510.15742/) (2025)
- [In-Context Learning with Unpaired Clips for Instruction-based Video Editing](https://mlsanity.com/paper/2510.14648/) (2025)
- [Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformers](https://mlsanity.com/paper/2506.03065/) (2025)
- [OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport](https://mlsanity.com/paper/2602.20205/) (2026)
- [Instruction-Based Video Editing by Repurposing an Image Editing Model](https://mlsanity.com/paper/2608.14790/) (2026)
- [GRNEdit: Efficient General Video Editing from a New Binary-Evidence Perspective in Generative Refinement Networks](https://mlsanity.com/paper/2608.16328/) (2026)
- [LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editing](https://mlsanity.com/paper/2604.17021/) (2026)

## Abstract (arXiv)

Video editing according to instructions is a highly challenging task due to the difficulty in collecting large-scale, high-quality edited video pair data. This scarcity not only limits the availability of training data but also hinders the systematic exploration of model architectures and training strategies. While prior work has improved specific aspects of video editing (e.g., synthesizing a video dataset using image editing techniques or decomposed video editing training), a holistic framework addressing the above challenges remains underexplored. In this study, we introduce InstructVEdit, a full-cycle instructional video editing approach that: (1) establishes a reliable dataset curation workflow to initialize training, (2) incorporates two model architectural improvements to enhance edit quality while preserving temporal consistency, and (3) proposes an iterative refinement strategy leveraging real-world data to enhance generalization and minimize train-test discrepancies. Extensive experiments show that InstructVEdit achieves state-of-the-art performance in instruction-based video editing, demonstrating robust adaptability to diverse real-world scenarios. Project page: https://o937-blip.github.io/InstructVEdit.