mlsanityData to 14 Oct 2026

DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors

arXiv 2310.12190 ↗First posted 18 Oct 2023cs.CVcited by 471

On the map: Diffusion models → Video generation

Abstract (arXiv)

Animating a still image offers an engaging visual experience. Traditional image animation techniques mainly focus on animating natural scenes with stochastic dynamics (e.g. clouds and fluid) or domain-specific motions (e.g. human hair or body motions), and thus limits their applicability to more general visual content. To overcome this limitation, we explore the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, we first project it into a text-aligned rich context representation space using a query transformer, which facilitates the video model to digest the image content in a compatible fashion. However, some visual details still struggle to be preserved in the resultant videos. To supplement with more precise image information, we further feed the full image to the diffusion model by concatenating it with the initial noises. Experimental results show that our proposed method can produce visually convincing and more logical & natural motions, as well as higher conformity to the input image. Comparative evaluation demonstrates the notable superiority of our approach over existing competitors.

Key facts. Cited by 471 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 79 of its 105 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

79 of 105 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  2. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  3. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation1 Sep
  4. 2023ModelScope Text-to-Video Technical Report12 Aug
  5. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  6. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  8. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Models24 Mar
  9. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  10. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  11. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  12. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
5 more
  1. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  3. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  4. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  5. 2022Video Diffusion Models7 Apr

Other references · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  3. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  4. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  6. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  8. 2020First Order Motion Model for Image Animationcited by 704
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  10. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  11. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  12. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  13. 2022MAGVIT: Masked Generative Video Transformercited by 346
  14. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  15. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  16. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  17. 2021Motion Representations for Articulated Animationcited by 255
  18. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  19. 2017Stochastic Variational Video Predictioncited by 234
  20. 2022Thin-Plate Spline Motion Model for Image Animationcited by 210
  21. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 208
  22. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 205
  23. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  24. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 175
  25. 2018Stochastic Adversarial Video Predictioncited by 173
  26. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  27. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  28. 2023ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Modelscited by 128
  29. 2020Stochastic Latent Residual Video Predictioncited by 108
  30. 2023Generative Image Dynamicscited by 94
  31. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 89
  32. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  33. 2020Improving Image Captioning with Better Use of Captionscited by 84
  34. 2020Improved Techniques for Training Single-Image GANscited by 78
  35. 2018Photo Wake-Up: 3D Character Animation from a Single Photocited by 60
  36. 2018Flow-Grounded Spatial-Temporal Video Prediction from Still Imagescited by 55
  37. 2020Animating Pictures with Eulerian Motion Fieldscited by 55
  38. 2023A Dynamic Multi-Scale Voxel Flow Network for Video Predictioncited by 53

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 139; 2025: 224; 2026: 100.

471 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  3. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  4. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  5. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  6. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  7. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  8. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  9. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 251
  10. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  11. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 211
  12. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
38 more
  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 185
  2. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 184
  3. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  4. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  5. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  6. 2025History-Guided Video Diffusioncited by 171
  7. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  8. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 159
  9. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 158
  10. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  11. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
  12. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  13. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 143
  14. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  15. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 132
  16. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  17. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  18. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 125
  19. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  20. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 124
  21. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 123
  22. 2024FasterCache: Training-Free Video Diffusion Model Acceleration with High Qualitycited by 122
  23. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 120
  24. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 117
  25. 2025Long Context Tuning for Video Generationcited by 113
  26. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 108
  27. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 98
  28. 2025Controllable Video Generation: A Surveycited by 98
  29. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 92
  30. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 89
  31. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 86
  32. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  33. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  34. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  35. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 76
  36. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  37. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 74
  38. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70

50 most cited of 471 All 471 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Controllable Longer Image Animation with Diffusion Models
  2. 2023Animate124: Animating One Image to 4D Dynamic Scene
  3. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  4. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidance
  5. 2024Diffusion Priors for Dynamic View Synthesis from Monocular Videos
  6. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising
  7. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation
  8. 2024Animated Stickers: Bringing Stickers to Life with Video Diffusion