# VideoDirector: Precise Video Editing via Text-to-Video Models

arXiv: https://arxiv.org/abs/2411.17592 · HTML: https://mlsanity.com/paper/2411.17592/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 32 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Latte: Latent Diffusion Transformer for Video Generation](https://mlsanity.com/paper/2401.03048/) (2024)
- [TokenFlow: Consistent Diffusion Features for Consistent Video Editing](https://mlsanity.com/paper/2307.10373/) (2023)
- [Understanding Diffusion Models: A Unified Perspective](https://mlsanity.com/paper/2208.11970/) (2022)
- [SVDiff: Compact Parameter Space for Diffusion Fine-Tuning](https://mlsanity.com/paper/2303.11305/) (2023)
- [Video-P2P: Video Editing with Cross-attention Control](https://mlsanity.com/paper/2303.04761/) (2023)
- [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023)
- [StableVideo: Text-driven Consistency-aware Diffusion Video Editing](https://mlsanity.com/paper/2308.09592/) (2023)
- [FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing](https://mlsanity.com/paper/2310.05922/) (2023)
- [RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models](https://mlsanity.com/paper/2312.04524/) (2023)
- [VidToMe: Video Token Merging for Zero-Shot Video Editing](https://mlsanity.com/paper/2312.10656/) (2023)
- [MotionClone: Training-Free Motion Cloning for Controllable Video Generation](https://mlsanity.com/paper/2406.05338/) (2024)
- [Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices](https://mlsanity.com/paper/2405.12211/) (2024)
- [TurboEdit: Instant text-based image editing](https://mlsanity.com/paper/2408.08332/) (2024)
- [AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing](https://mlsanity.com/paper/2312.08019/) (2023)

## Built on by (most cited first, 30 of 36)

- [Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset](https://mlsanity.com/paper/2510.15742/) (2025)
- [In-Context Learning with Unpaired Clips for Instruction-based Video Editing](https://mlsanity.com/paper/2510.14648/) (2025)
- [FlowDirector: Training-Free Flow Steering for Precise Text-to-Video Editing](https://mlsanity.com/paper/2506.05046/) (2025)
- [VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization](https://mlsanity.com/paper/2512.16906/) (2025)
- [TC-Light: Temporally Coherent Generative Rendering for Realistic World Transfer](https://mlsanity.com/paper/2506.18904/) (2025)
- [PISCO: Precise Video Instance Insertion with Sparse Control](https://mlsanity.com/paper/2602.08277/) (2026)
- [Lynx: Towards High-Fidelity Personalized Video Generation](https://mlsanity.com/paper/2509.15496/) (2025)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [Causally Steered Diffusion for Automated Video Counterfactual Generation](https://mlsanity.com/paper/2506.14404/) (2025)
- [ASTRA: Let Arbitrary Subjects Transform in Video Editing](https://mlsanity.com/paper/2510.01186/) (2025)
- [VideoAgent: All-in-One Framework for Video Understanding and Editing](https://mlsanity.com/paper/2606.23327/) (2026)
- [EditCast3D: Single-Frame-Guided 3D Editing with Video Propagation and View Selection](https://mlsanity.com/paper/2510.13652/) (2025)
- [FFP-300K: Scaling First-Frame Propagation for Generalizable Video Editing](https://mlsanity.com/paper/2601.01720/) (2026)
- [DFVEdit: Conditional Delta Flow Vector for Zero-shot Video Editing](https://mlsanity.com/paper/2506.20967/) (2025)
- [Versatile Editing of Video Content, Actions, and Dynamics without Training](https://mlsanity.com/paper/2603.17989/) (2026)
- [Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework](https://mlsanity.com/paper/2605.23891/) (2026)
- [ReBind: Multi-Reference Video Editing via Structured Instructions with Explicit Reference Relationships](https://mlsanity.com/paper/2607.14681/) (2026)
- [Vectorized Video Representation with Easy Editing via Hierarchical Spatio-Temporally Consistent Proxy Embedding](https://mlsanity.com/paper/2510.12256/) (2025)
- [Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting](https://mlsanity.com/paper/2604.21776/) (2026)
- [StreamEdit: Training-Free Video Editing via Few-Step Streaming Video Generation](https://mlsanity.com/paper/2605.21466/) (2026)
- [GRNEdit: Efficient General Video Editing from a New Binary-Evidence Perspective in Generative Refinement Networks](https://mlsanity.com/paper/2608.16328/) (2026)
- [RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editing](https://mlsanity.com/paper/2602.06871/) (2026)
- [Preserve, Reveal, Expand: Faithful 4D Video Editing with Region-Aware Conditioning](https://mlsanity.com/paper/2605.20961/) (2026)
- [Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis](https://mlsanity.com/paper/2606.29020/) (2026)
- [The 1st Solution for MOSEv2 Challenge 2025: Long-term and Concept-aware Video Segmentation via SeC](https://mlsanity.com/paper/2509.19183/) (2025)
- [Beyond Rigid: Benchmarking Non-Rigid Video Editing](https://mlsanity.com/paper/2601.18340/) (2026)
- [When to Lock Attention: Training-Free KV Control in Video Diffusion](https://mlsanity.com/paper/2603.09657/) (2026)
- [Re-Prompting SAM 3 via Object Retrieval: 3rd of the 5th PVUW MOSE Track](https://mlsanity.com/paper/2603.23788/) (2026)
- [Sound Sparks Motion: Audio and Text Tuning for Video Editing](https://mlsanity.com/paper/2605.15307/) (2026)
- [Occlusion-Aware Physics-Semantic Keyframe Selection for Robust Video Editing](https://mlsanity.com/paper/2605.23192/) (2026)

## Abstract (arXiv)

Despite the typical inversion-then-editing paradigm using text-to-image (T2I) models has demonstrated promising results, directly extending it to text-to-video (T2V) models still suffers severe artifacts such as color flickering and content distortion. Consequently, current video editing methods primarily rely on T2I models, which inherently lack temporal-coherence generative ability, often resulting in inferior editing results. In this paper, we attribute the failure of the typical editing paradigm to: 1) Tightly Spatial-temporal Coupling. The vanilla pivotal-based inversion strategy struggles to disentangle spatial-temporal information in the video diffusion model; 2) Complicated Spatial-temporal Layout. The vanilla cross-attention control is deficient in preserving the unedited content. To address these limitations, we propose a spatial-temporal decoupled guidance (STDG) and multi-frame null-text optimization strategy to provide pivotal temporal cues for more precise pivotal inversion. Furthermore, we introduce a self-attention control strategy to maintain higher fidelity for precise partial content editing. Experimental results demonstrate that our method (termed VideoDirector) effectively harnesses the powerful temporal generation capabilities of T2V models, producing edited videos with state-of-the-art performance in accuracy, motion smoothness, realism, and fidelity to unedited content.