mlsanityData to 14 Oct 2026

Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets

arXiv 2311.15127 ↗First posted 25 Nov 2023cs.CVcited by 2,581

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. Recently, latent diffusion models trained for 2D image synthesis have been turned into generative video models by inserting temporal layers and finetuning them on small, high-quality video datasets. However, training methods in the literature vary widely, and the field has yet to agree on a unified strategy for curating video data. In this paper, we identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning. Furthermore, we demonstrate the necessity of a well-curated pretraining dataset for generating high-quality videos and present a systematic curation process to train a strong base model, including captioning and filtering strategies. We then explore the impact of finetuning our base model on high-quality data and train a text-to-video model that is competitive with closed-source video generation. We also show that our base model provides a powerful motion representation for downstream tasks such as image-to-video generation and adaptability to camera motion-specific LoRA modules. Finally, we demonstrate that our model provides a strong multi-view 3D-prior and can serve as a base to finetune a multi-view diffusion model that jointly generates multiple views of objects in a feedforward fashion, outperforming image-based methods at a fraction of their compute budget. We release code and model weights at https://github.com/Stability-AI/generative-models .

Key facts. Cited by 2,581 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 88 of its 104 references that are in the map. First posted on arXiv 25 Nov 2023.

Builds on

88 of 104 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
  3. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 83

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2022Video Diffusion Modelscited by 2,305
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  10. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  11. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  12. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  13. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  14. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  16. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  17. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  18. 2021Variational Diffusion Modelscited by 1,153
  19. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  20. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  21. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  22. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  23. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  24. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  25. 2022On Distillation of Guided Diffusion Modelscited by 725
  26. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 699
  27. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  28. 2023ModelScope Text-to-Video Technical Reportcited by 658
  29. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 645
  30. 2021Score-based Generative Modeling in Latent Spacecited by 636
  31. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
  32. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  33. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 465
  34. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  35. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  36. 2016Generating Videos with Scene Dynamicscited by 376
  37. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  38. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 661; 2025: 1037; 2026: 859.

2,581 papers.

Sort:

By area:

  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  2. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  4. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  5. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  6. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  7. 2024Genie: Generative Interactive Environmentscited by 747
  8. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 684
  9. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  10. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  11. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  12. 2024LTX-Video: Realtime Video Latent Diffusioncited by 581
38 more
  1. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  2. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  3. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  4. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  5. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  6. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  7. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 433
  8. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  9. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  10. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  11. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  12. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  13. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  14. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  15. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  16. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  17. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  18. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  19. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  20. 2024Discrete Flow Matchingcited by 317
  21. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  22. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 305
  23. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  24. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  25. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 287
  26. 2024DepthSplat: Connecting Gaussian Splatting and Depthcited by 286
  27. 2025Improving Video Generation with Human Feedbackcited by 284
  28. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 283
  29. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 279
  30. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  31. 2024Navigation World Modelscited by 272
  32. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 269
  33. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 266
  34. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 251
  35. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  36. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  37. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  38. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 234

50 most cited of 2,581 All 2,581 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition
  2. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  3. 2023Video Probabilistic Diffusion Models in Projected Latent Space
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models
  5. 2023MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance
  6. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
  7. 2026Efficient Video Diffusion Models: Advancements and Challenges
  8. 2023LLM-grounded Video Diffusion Models