mlsanityData to 14 Oct 2026

VideoCrafter1: Open Diffusion Models for High-Quality Video Generation

arXiv 2310.19512 ↗First posted 30 Oct 2023cs.CVcited by 535

On the map: Diffusion models → Video generation

Abstract (arXiv)

Video generation has increasingly gained interest in both academia and industry. Although commercial tools can generate plausible videos, there is a limited number of open-source models available for researchers and engineers. In this work, we introduce two diffusion models for high-quality video generation, namely text-to-video (T2V) and image-to-video (I2V) models. T2V models synthesize a video based on a given text input, while I2V models incorporate an additional image input. Our proposed T2V model can generate realistic and cinematic-quality videos with a resolution of 1024 × 576, outperforming other open-source T2V models in terms of quality. The I2V model is designed to produce videos that strictly adhere to the content of the provided reference image, preserving its content, structure, and style. This model is the first open-source I2V foundation model capable of transforming a given image into a video clip while maintaining content preservation constraints. We believe that these open-source video generation models will contribute significantly to the technological advancements within the community.

Key facts. Cited by 535 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 58 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

51 of 58 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  2. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation1 Sep
  3. 2023ModelScope Text-to-Video Technical Report12 Aug
  4. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  5. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  6. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  8. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  9. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  10. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  11. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  12. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
2 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  2. 2022Video Diffusion Models7 Apr

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
25 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  4. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  8. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  9. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  10. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  11. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  12. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  13. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  14. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  15. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  16. 2022MAGVIT: Masked Generative Video Transformercited by 346
  17. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  18. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  19. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  20. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  21. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  22. 2023ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Modelscited by 128
  23. 2023Seer: Language Instructed Video Prediction with Latent Diffusion Modelscited by 52
  24. 2023Real-World Image Variation by Aligning Diffusion Inversion Chaincited by 37
  25. 2023Diffusion in the Dark: A Diffusion Model for Low-Light Text Recognitioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 190; 2025: 217; 2026: 104.

535 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  3. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  5. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  6. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  7. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  8. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  9. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  10. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  11. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 269
  12. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
38 more
  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  2. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
  3. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 202
  4. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 198
  5. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  6. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  7. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 184
  8. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
  9. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  10. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 168
  11. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 166
  12. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 164
  13. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  14. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  15. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
  16. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  17. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  18. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  19. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  20. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 137
  21. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  22. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 132
  23. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 132
  24. 2025Fast Video Generation with Sliding Tile Attentioncited by 128
  25. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 125
  26. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 120
  27. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 117
  28. 2023Animate124: Animating One Image to 4D Dynamic Scenecited by 115
  29. 2025Long Context Tuning for Video Generationcited by 113
  30. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 105
  31. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 102
  32. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 99
  33. 2025Controllable Video Generation: A Surveycited by 98
  34. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 97
  35. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  36. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
  37. 2023FreeInit: Bridging Initialization Gap in Video Diffusion Modelscited by 95
  38. 2025Mixture of Contexts for Long Video Generationcited by 95

50 most cited of 535 All 535 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024S2DM: Sector-Shaped Diffusion Models for Video Generation
  2. 2026UniVid: Pyramid Diffusion Model for High Quality Video Generation
  3. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
  4. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text
  5. 2024Grid Diffusion Models for Text-to-Video Generation
  6. 2026Efficient Video Diffusion Models: Advancements and Challenges
  7. 2024SF-V: Single Forward Video Generation Model
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets