mlsanityData to 7 Oct 2026

Dreamix: Video Diffusion Models are General Video Editors

arXiv 2302.01329 ↗First posted 2 Feb 2023cs.CVcited by 189

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-driven image and video diffusion models have recently achieved unprecedented generation realism. While diffusion models have been successfully applied for image editing, very few works have done so for video editing. We present the first diffusion-based method that is able to perform text-based motion and appearance editing of general videos. Our approach uses a video diffusion model to combine, at inference time, the low-resolution spatio-temporal information from the original video with new, high resolution information that it synthesized to align with the guiding text prompt. As obtaining high-fidelity to the original video requires retaining some of its high-resolution information, we add a preliminary stage of finetuning the model on the original video, significantly boosting fidelity. We propose to improve motion editability by a new, mixed objective that jointly finetunes with full temporal attention and with temporal attention masking. We further introduce a new framework for image animation. We first transform the image into a coarse video by simple image processing operations such as replication and perspective geometric projections, and then use our general video editor to animate it. As a further application, we can use our method for subject-driven video generation. Extensive qualitative and numerical experiments showcase the remarkable editing ability of our method and establish its superior performance compared to baseline methods.

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 43 of its 51 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

43 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  3. 2022Video Diffusion Models7 Apr

Other references · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
28 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  8. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  9. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  10. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  12. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  13. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  14. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  15. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  16. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  17. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  18. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  19. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  20. 2022Blended Latent Diffusioncited by 430
  21. 2022MAGVIT: Masked Generative Video Transformercited by 341
  22. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  23. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  24. 2021Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contractioncited by 306
  25. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  26. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  27. 2022Conffusion: Confidence Intervals for Diffusion Modelscited by 33
  28. 2020Image Shape Manipulation from a Single Augmented Training Samplecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 87; 2024: 53; 2025: 33; 2026: 16.

189 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  4. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  6. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  7. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  8. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  9. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  10. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  11. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  12. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
38 more
  1. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  2. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  3. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  4. 2025Video models are zero-shot learners and reasonerscited by 208
  5. 2023A Survey on Video Diffusion Modelscited by 203
  6. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  7. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  8. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  9. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  10. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  11. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  12. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  13. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  14. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  15. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  16. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  17. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  18. 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 119
  19. 2023RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Modelscited by 113
  20. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  21. 2023VideoLCM: Video Latent Consistency Modelcited by 90
  22. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  23. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 81
  24. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Modelscited by 80
  25. 2023Consistent Video-to-Video Transfer Using Synthetic Datasetcited by 79
  26. 2023Face0: Instantaneously Conditioning a Text-to-Image Model on a Facecited by 77
  27. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generationcited by 74
  28. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondencecited by 69
  29. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 68
  30. 2023Edit-A-Video: Single Video Editing with Object-Aware Consistencycited by 65
  31. 2023InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructionscited by 64
  32. 2023The Chosen One: Consistent Characters in Text-to-Image Diffusion Modelscited by 62
  33. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  34. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generationcited by 59
  35. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  36. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  37. 2024Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Modelscited by 55
  38. 2023CVPR 2023 Text Guided Video Editing Competitioncited by 54

50 most cited of 189 All 189 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
  2. 2023MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance
  3. 2023Structure and Content-Guided Video Synthesis with Diffusion Models
  4. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  6. 2024Temporally Consistent Object Editing in Videos using Extended Attention
  7. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
  8. 2024Animate Your Motion: Turning Still Images into Dynamic Videos