mlsanityData to 7 Oct 2026

VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models

arXiv 2401.09047 ↗First posted 17 Jan 2024cs.CVcited by 568

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-to-video generation aims to produce a video based on a given prompt. Recently, several commercial video models have been able to generate plausible videos with minimal noise, excellent details, and high aesthetic scores. However, these models rely on large-scale, well-filtered, high-quality videos that are not accessible to the community. Many existing research works, which train models using the low-quality WebVid-10M dataset, struggle to generate high-quality videos because the models are optimized to fit WebVid-10M. In this work, we explore the training scheme of video models extended from Stable Diffusion and investigate the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model. We first analyze the connection between the spatial and temporal modules of video models and the distribution shift to low-quality videos. We observe that full training of all modules results in a stronger coupling between spatial and temporal modules than only training temporal modules. Based on this stronger coupling, we shift the distribution to higher quality without motion degradation by finetuning spatial modules with high-quality images, resulting in a generic high-quality video model. Evaluations are conducted to demonstrate the superiority of the proposed method, particularly in picture quality, motion, and concept composition.

Key facts. Cited by 568 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 50 of its 54 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

50 of 54 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  2. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  3. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  5. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  6. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  7. 2023ModelScope Text-to-Video Technical Report12 Aug
  8. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  9. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  11. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  12. 2023Video Probabilistic Diffusion Models in Projected Latent Space15 Feb
5 more
  1. 2022VIDM: Video Implicit Diffusion Models1 Dec
  2. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  4. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  5. 2022Video Diffusion Models7 Apr

Other references · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
21 more
  1. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  2. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  3. 2017Generative Adversarial Networks: An Overviewcited by 588
  4. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  5. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  6. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  7. 2016Tutorial on Variational Autoencoderscited by 404
  8. 2016Generating Videos with Scene Dynamicscited by 376
  9. 2022MAGVIT: Masked Generative Video Transformercited by 341
  10. 2022Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectivescited by 314
  11. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  12. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  13. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  14. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  15. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  16. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  17. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  18. 2023ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Modelscited by 127
  19. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  20. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
  21. 2023Real-World Image Variation by Aligning Diffusion Inversion Chaincited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 175; 2025: 249; 2026: 144.

568 papers.

Sort:

By area:

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  4. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  6. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  7. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  8. 2025Improving Video Generation with Human Feedbackcited by 267
  9. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 245
  10. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  11. 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 202
  12. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
38 more
  1. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  2. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  3. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  4. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  5. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  6. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  7. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  8. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  9. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  10. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  11. 2024FIFO-Diffusion: Generating Infinite Videos from Text without Trainingcited by 134
  12. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  13. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  14. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  15. 2025Long Context Tuning for Video Generationcited by 110
  16. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  17. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 102
  18. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  19. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  20. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 100
  21. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  22. 2025Mixture of Contexts for Long Video Generationcited by 90
  23. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  24. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  25. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  26. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  27. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  28. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 84
  29. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  30. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
  31. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjectscited by 76
  32. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  33. 2024GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Controlcited by 76
  34. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  35. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  36. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generationcited by 73
  37. 2024Sonic: Shifting Focus to Global Audio Perception in Portrait Animationcited by 70
  38. 2024CV-VAE: A Compatible Video VAE for Latent Generative Video Modelscited by 68

50 most cited of 568 All 568 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  2. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  3. 2024VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models
  4. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generation
  5. 2026Efficient Video Diffusion Models: Advancements and Challenges
  6. 2024Grid Diffusion Models for Text-to-Video Generation
  7. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation
  8. 2024JVID: Joint Video-Image Diffusion for Visual-Quality and Temporal-Consistency in Video Generation