mlsanityData to 7 Oct 2026

LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models

arXiv 2309.15103 ↗First posted 26 Sep 2023cs.CVcited by 349

On the map: Diffusion models → Video generation

Abstract (arXiv)

This work aims to learn a high-quality text-to-video (T2V) generative model by leveraging a pre-trained text-to-image (T2I) model as a basis. It is a highly desirable yet challenging task to simultaneously a) accomplish the synthesis of visually realistic and temporally coherent videos while b) preserving the strong creative generation nature of the pre-trained T2I model. To this end, we propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models, comprising a base T2V model, a temporal interpolation model, and a video super-resolution model. Our key insights are two-fold: 1) We reveal that the incorporation of simple temporal self-attentions, coupled with rotary positional encoding, adequately captures the temporal correlations inherent in video data. 2) Additionally, we validate that the process of joint image-video fine-tuning plays a pivotal role in producing high-quality and creative outcomes. To enhance the performance of LaVie, we contribute a comprehensive and diverse video dataset named Vimeo25M, consisting of 25 million text-video pairs that prioritize quality, diversity, and aesthetic appeal. Extensive experiments demonstrate that LaVie achieves state-of-the-art performance both quantitatively and qualitatively. Furthermore, we showcase the versatility of pre-trained LaVie models in various long video generation and personalized video synthesis applications.

Key facts. Cited by 349 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 64 of its 89 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

64 of 89 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023ModelScope Text-to-Video Technical Report12 Aug
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  3. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  6. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  8. 2022Video Diffusion Models7 Apr

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2017Neural Discrete Representation Learningcited by 4,650
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  10. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  13. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  14. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  15. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  16. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  17. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  18. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  19. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  20. 2023Zero-shot Image-to-Image Translationcited by 503
  21. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  22. 2016Generating Videos with Scene Dynamicscited by 376
  23. 2021BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignmentcited by 340
  24. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  25. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  26. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  27. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  28. 2022Towards Robust Blind Face Restoration with Codebook Lookup Transformercited by 230
  29. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  30. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  31. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  32. 2021Investigating Tradeoffs in Real-World Video Super-Resolutioncited by 133
  33. 2018Disentangled Sequential Autoencodercited by 117
  34. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  35. 2022Generating Long Videos of Dynamic Scenescited by 113
  36. 2020Cross-Scale Internal Graph Neural Network for Image Super-Resolutioncited by 94
  37. 2021Robust Reference-based Super-Resolution via C2-Matchingcited by 68
  38. 2023Text2Performer: Text-Driven Human Video Generationcited by 53

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 38; 2024: 142; 2025: 111; 2026: 58.

349 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  5. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  6. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  7. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  8. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  9. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  10. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  11. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  12. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
38 more
  1. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  2. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  3. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  4. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  5. 2025LongLive: Real-time Interactive Long Video Generationcited by 237
  6. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Predictioncited by 231
  7. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  8. 2023A Survey on Video Diffusion Modelscited by 203
  9. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 191
  10. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  11. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  12. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  13. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  14. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  15. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  16. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  17. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  18. 2025Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Modelscited by 160
  19. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  20. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  21. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  22. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  23. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  24. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  25. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  26. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  27. 2025Fast Video Generation with Sliding Tile Attentioncited by 123
  28. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  29. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  30. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  31. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  32. 2023FreeInit: Bridging Initialization Gap in Video Diffusion Modelscited by 93
  33. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  34. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 92
  35. 2024FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attentioncited by 87
  36. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Modelscited by 86
  37. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  38. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85

50 most cited of 349 All 349 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  2. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation
  3. 2024Latte: Latent Diffusion Transformer for Video Generation
  4. 2025CascadeV: An Implementation of Wurstchen Architecture for Video Generation
  5. 2023Photorealistic Video Generation with Diffusion Models
  6. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  7. 2024Lumiere: A Space-Time Diffusion Model for Video Generation
  8. 2024VideoTetris: Towards Compositional Text-to-Video Generation