mlsanityData to 14 Oct 2026

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

arXiv 2304.08818 ↗First posted 18 Apr 2023cs.CV cs.LGcited by 1,476

On the map: Diffusion models → Video generation

Abstract (arXiv)

Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion model in a compressed lower-dimensional latent space. Here, we apply the LDM paradigm to high-resolution video generation, a particularly resource-intensive task. We first pre-train an LDM on images only; then, we turn the image generator into a video generator by introducing a temporal dimension to the latent space diffusion model and fine-tuning on encoded image sequences, i.e., videos. Similarly, we temporally align diffusion model upsamplers, turning them into temporally consistent video super resolution models. We focus on two relevant real-world applications: Simulation of in-the-wild driving data and creative content creation with text-to-video modeling. In particular, we validate our Video LDM on real driving videos of resolution 512 x 1024, achieving state-of-the-art performance. Furthermore, our approach can easily leverage off-the-shelf pre-trained image LDMs, as we only need to train a temporal alignment model in that case. Doing so, we turn the publicly available, state-of-the-art text-to-image LDM Stable Diffusion into an efficient and expressive text-to-video model with resolution up to 1280 x 2048. We show that the temporal layers trained in this way generalize to different fine-tuned text-to-image LDMs. Utilizing this property, we show the first results for personalized text-to-video generation, opening exciting directions for future content creation. Project page: https://research.nvidia.com/labs/toronto-ai/VideoLDM/

Key facts. Cited by 1,476 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 95 of its 121 references that are in the map. First posted on arXiv 18 Apr 2023.

Builds on

95 of 121 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  3. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  5. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  6. 2022Flexible Diffusion Modeling of Long Videos23 May
  7. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  8. 2022Video Diffusion Models7 Apr
  9. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 86

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  12. 2022Classifier-Free Diffusion Guidancecited by 5,847
38 more
  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2016Improved Techniques for Training GANscited by 2,970
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  13. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  14. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  15. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  16. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  17. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  18. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  19. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  20. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  21. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  22. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  23. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  24. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  25. 2022Denoising Diffusion Restoration Modelscited by 963
  26. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  27. 2022On Distillation of Guided Diffusion Modelscited by 725
  28. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  29. 2018Which Training Methods for GANs do actually Converge?cited by 679
  30. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  31. 2021Score-based Generative Modeling in Latent Spacecited by 636
  32. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  33. 2022Fast Sampling of Diffusion Models with Exponential Integratorcited by 532
  34. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 516
  35. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 492
  36. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  37. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  38. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 215; 2024: 515; 2025: 479; 2026: 267.

1,476 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  6. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 938
  7. 2023Adversarial Diffusion Distillationcited by 759
  8. 2024Genie: Generative Interactive Environmentscited by 747
  9. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  10. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  11. 2023ModelScope Text-to-Video Technical Reportcited by 658
  12. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
38 more
  1. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  2. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  3. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  5. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 546
  6. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  7. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  8. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  9. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  10. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  11. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  12. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  13. 2023Learning Interactive Real-World Simulatorscited by 458
  14. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  15. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  16. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  17. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  18. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  19. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  20. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  21. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  22. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  23. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  24. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  25. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  26. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  27. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  28. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  29. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  30. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 302
  31. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  32. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  33. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  34. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 279
  35. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 266
  36. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  37. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  38. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Modelscited by 254

50 most cited of 1,476 All 1,476 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation
  2. 2024Pixel-Space Post-Training of Latent Diffusion Models
  3. 2023Video Probabilistic Diffusion Models in Projected Latent Space
  4. 2024RepLDM: Reprogramming Pretrained Latent Diffusion Models for High-Quality, High-Efficiency, High-Resolution Image Generation
  5. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  6. 2023Fuse Your Latents: Video Editing with Multi-source Latent Diffusion Models
  7. 2026LDM-is-AE: Latent Diffusion Model is an Auto-Encoder for End-to-End Image Generation
  8. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference