mlsanityData to 7 Oct 2026

Diffusion Model-Based Video Editing: A Survey

arXiv 2407.07111 ↗First posted 26 Jun 2024cs.CV cs.AI cs.LGcited by 44

On the map: Diffusion models → Video generation

Abstract (arXiv)

The rapid development of diffusion models (DMs) has significantly advanced image and video applications, making "what you want is what you see" a reality. Among these, video editing has gained substantial attention and seen a swift rise in research activity, necessitating a comprehensive and systematic review of the existing literature. This paper reviews diffusion model-based video editing techniques, including theoretical foundations and practical applications. We begin by overviewing the mathematical formulation and image domain's key methods. Subsequently, we categorize video editing approaches by the inherent connections of their core technologies, depicting evolutionary trajectory. This paper also dives into novel applications, including point-based editing and pose-guided human video editing. Additionally, we present a comprehensive comparison using our newly introduced V2VBench. Building on the progress achieved to date, the paper concludes with ongoing challenges and potential directions for future research.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 223 of its 244 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

223 of 244 references.

Closest ancestors · 54

Same direction, newest first

  1. 2024FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation19 Mar
  2. 2024Video Editing via Factorized Diffusion Distillation14 Mar
  3. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models10 Mar
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  5. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  6. 2023FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis29 Dec
  7. 2023Fairy: Fast Parallelized Instruction-Guided Video-to-Video Synthesis20 Dec
  8. 2023MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers19 Dec
  9. 2023VidToMe: Video Token Merging for Zero-Shot Video Editing17 Dec
  10. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models7 Dec
  11. 2023DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing5 Dec
  12. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence4 Dec
38 more
  1. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models1 Dec
  2. 2023MotionEditor: Editing Video Motion via Content-Aware Diffusion30 Nov
  3. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation28 Nov
  4. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model27 Nov
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  6. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  7. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  8. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  9. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  10. 2023A Survey on Video Diffusion Models16 Oct
  11. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  12. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing9 Oct
  13. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models2 Oct
  14. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Models28 Sep
  15. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  16. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  17. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing18 Aug
  18. 2023ModelScope Text-to-Video Technical Report12 Aug
  19. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet26 Jul
  20. 2023AnyDoor: Zero-shot Object-level Image Customization18 Jul
  21. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  22. 2023VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing14 Jun
  23. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  24. 2023Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models30 May
  25. 2023Towards Consistent Video Editing with Text-to-Image Diffusion Models27 May
  26. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  27. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  28. 2023Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts15 May
  29. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis27 Apr
  30. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  31. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  32. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  33. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Models24 Mar
  34. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  35. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  36. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  37. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  38. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb

50 newest of 54 All 54 (JSON)

Other references · 169

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  13. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  14. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  15. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  16. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  17. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  18. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  19. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  20. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  21. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  22. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  23. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  24. 2015Variational Inference with Normalizing Flowscited by 2,161
  25. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  26. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  27. 2023Consistency Modelscited by 1,778
  28. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  29. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  30. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  31. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  32. 2018Group Normalizationcited by 1,514
  33. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  34. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  35. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  36. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  37. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  38. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175

50 most cited of 169 All 169 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 23; 2026: 13.

44 papers.

Sort:

By area:

  1. 2025Video models are zero-shot learners and reasonerscited by 208
  2. 2024Taming Rectified Flow for Inversion and Editingcited by 200
  3. 2025Controllable Video Generation: A Surveycited by 93
  4. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 83
  5. 2025InsViE-1M: Effective Instruction-based Video Editing with Elaborate Dataset Constructioncited by 62
  6. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 53
  7. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  8. 2025Generative Physical AI in Vision: A Surveycited by 24
  9. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  10. 2024AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restorationcited by 15
  11. 2024Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmarkcited by 13
  12. 2025Get In Video: Add Anything You Want to the Videocited by 12
32 more
  1. 2024SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editingcited by 11
  2. 2024EditBoard: Towards a Comprehensive Evaluation Benchmark for Text-Based Video Editing Modelscited by 10
  3. 2025ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answeringcited by 8
  4. 2025VEU-Bench: Towards Comprehensive Understanding of Video Editingcited by 7
  5. 2026AgenticVBench: Can AI Agents Complete Real-World Post-Production Tasks?cited by 7
  6. 2026Evolution of Video Generative Foundationscited by 6
  7. 2025AdaFlow: Efficient Long Video Editing via Adaptive Attention Slimming And Keyframe Selectioncited by 5
  8. 2025Causally Steered Diffusion for Automated Video Counterfactual Generationcited by 5
  9. 2025Evolutionary Caching to Accelerate Your Off-the-Shelf Diffusion Modelcited by 5
  10. 2025Taming Flow-based I2V Models for Creative Video Editingcited by 5
  11. 2025VideoCoF: Unified Video Editing with Temporal Reasonercited by 5
  12. 2024Replace Anyone in Videoscited by 4
  13. 2024Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuningcited by 2
  14. 2025DAPE: Dual-Stage Parameter-Efficient Fine-Tuning for Consistent Video Editing with Diffusion Modelscited by 2
  15. 2025Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Surveycited by 2
  16. 2025LoVoRA: Text-guided and Mask-free Video Object Removal and Addition with Learnable Object-aware Localizationcited by 2
  17. 2025Semantix: An Energy Guided Sampler for Semantic Style Transfercited by 1
  18. 2025Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representationscited by 1
  19. 2026RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editingcited by 1
  20. 2026ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformercited by 1
  21. 2026ActivityForensics: A Comprehensive Benchmark for Localizing Manipulated Activity in Videoscited by 1
  22. 2026LIVE: Leveraging Image Manipulation Priors for Instruction-based Video Editingcited by 1
  23. 2026Image-to-Video Diffusion: From Foundations to Open Frontierscited by 1
  24. 2026MSEditor: Toward Consistent Multi-Shot Video Editingcited by 1
  25. 2025BounTCHA: A CAPTCHA Utilizing Boundary Identification in Guided Generative AI-extended Videoscited by 0
  26. 2025Generating time-consistent dynamics with discriminator-guided image diffusion modelscited by 0
  27. 2025Lightning Fast Caching-based Parallel Denoising Prediction for Accelerating Talking Head Generationcited by 0
  28. 2026Beyond Rigid: Benchmarking Non-Rigid Video Editingcited by 0
  29. 2026StreamWise: Serving Multi-Modal Generation in Real-Time at Scalecited by 0
  30. 2026TeleMorpher: Toward Robust Simultaneous Motion-Location Editingcited by 0
  31. 2026VicEdit: Learning to Edit Videos from Visual In-Context Examplescited by 0
  32. 2026ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Video Diffusion Models: A Survey
  2. 2025Image Editing with Diffusion Models: A Survey
  3. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models
  4. 2026Efficient Video Diffusion Models: Advancements and Challenges
  5. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  6. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models
  7. 2025FADE: Frequency-Aware Diffusion Model Factorization for Video Editing
  8. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models