# VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing

arXiv: https://arxiv.org/abs/2306.08707 · HTML: https://mlsanity.com/paper/2306.08707/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 44 of its 45 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)

## Built on by (most cited first, 30 of 36)

- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence](https://mlsanity.com/paper/2312.02087/) (2023)
- [UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing](https://mlsanity.com/paper/2402.13185/) (2024)
- [Mora: Enabling Generalist Video Generation via A Multi-Agent Framework](https://mlsanity.com/paper/2403.13248/) (2024)
- [Diffusion Model-Based Video Editing: A Survey](https://mlsanity.com/paper/2407.07111/) (2024)
- [ViViD: Video Virtual Try-on using Diffusion Models](https://mlsanity.com/paper/2405.11794/) (2024)
- [TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models](https://mlsanity.com/paper/2403.17005/) (2024)
- [MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion](https://mlsanity.com/paper/2405.20325/) (2024)
- [Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion](https://mlsanity.com/paper/2403.14617/) (2024)
- [DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut](https://mlsanity.com/paper/2406.02842/) (2024)
- [DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing](https://mlsanity.com/paper/2310.10624/) (2023)
- [DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes](https://mlsanity.com/paper/2412.19458/) (2024)
- [MagicProp: Diffusion-based Video Editing via Motion-aware Appearance Propagation](https://mlsanity.com/paper/2309.00908/) (2023)
- [VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization](https://mlsanity.com/paper/2512.16906/) (2025)
- [Dancing Avatar: Pose and Text-Guided Human Motion Videos Synthesis with Image Diffusion Model](https://mlsanity.com/paper/2308.07749/) (2023)
- [EasyV2V: A High-quality Instruction-based Video Editing Framework](https://mlsanity.com/paper/2512.16920/) (2025)
- [Visual Prompting for One-shot Controllable Video Editing without Inversion](https://mlsanity.com/paper/2504.14335/) (2025)
- [SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing](https://mlsanity.com/paper/2603.19228/) (2026)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing](https://mlsanity.com/paper/2312.03772/) (2023)
- [EVE: Efficient zero-shot text-based Video Editing with Depth Map Guidance and Temporal Consistency Constraints](https://mlsanity.com/paper/2308.10648/) (2023)
- [GVDIFF: Grounded Text-to-Video Generation with Diffusion Models](https://mlsanity.com/paper/2407.01921/) (2024)
- [PickStyle: Video-to-Video Style Transfer with Context-Style Adapters](https://mlsanity.com/paper/2510.07546/) (2025)
- [RealCraft: Attention Control as A Tool for Zero-Shot Consistent Video Editing](https://mlsanity.com/paper/2312.12635/) (2023)
- [Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuning](https://mlsanity.com/paper/2412.02114/) (2024)
- [NeoMap: Training-free Novel-View Synthesis from Single Images and Videos](https://mlsanity.com/paper/2607.01962/) (2026)
- [EgoPlay: Event-Triggered Video Editing for Egocentric Streams](https://mlsanity.com/paper/2607.24560/) (2026)
- [Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications](https://mlsanity.com/paper/2410.02952/) (2024)

## Abstract (arXiv)

Recently, diffusion-based generative models have achieved remarkable success for image generation and edition. However, existing diffusion-based video editing approaches lack the ability to offer precise control over generated content that maintains temporal consistency in long-term videos. On the other hand, atlas-based methods provide strong temporal consistency but are costly to edit a video and lack spatial control. In this work, we introduce VidEdit, a novel method for zero-shot text-based video editing that guarantees robust temporal and spatial consistency. In particular, we combine an atlas-based video representation with a pre-trained text-to-image diffusion model to provide a training-free and efficient video editing method, which by design fulfills temporal smoothness. To grant precise user control over generated content, we utilize conditional information extracted from off-the-shelf panoptic segmenters and edge detectors which guides the diffusion sampling process. This method ensures a fine spatial control on targeted regions while strictly preserving the structure of the original video. Our quantitative and qualitative experiments show that VidEdit outperforms state-of-the-art methods on DAVIS dataset, regarding semantic faithfulness, image preservation, and temporal consistency metrics. With this framework, processing a single video only takes approximately one minute, and it can generate multiple compatible edits based on a unique text prompt. Project web-page at https://videdit.github.io