mlsanityData to 14 Oct 2026

VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation

arXiv 2309.00398 ↗First posted 1 Sep 2023cs.CV cs.MMcited by 74

On the map: Diffusion models → Video generation

Abstract (arXiv)

In this paper, we present VideoGen, a text-to-video generation approach, which can generate a high-definition video with high frame fidelity and strong temporal consistency using reference-guided latent diffusion. We leverage an off-the-shelf text-to-image generation model, e.g., Stable Diffusion, to generate an image with high content quality from the text prompt, as a reference image to guide video generation. Then, we introduce an efficient cascaded latent diffusion module conditioned on both the reference image and the text prompt, for generating latent video representations, followed by a flow-based temporal upsampling step to improve the temporal resolution. Finally, we map latent video representations into a high-definition video through an enhanced video decoder. During training, we use the first frame of a ground-truth video as the reference image for training the cascaded latent diffusion module. The main characterises of our approach include: the reference image generated by the text-to-image model improves the visual fidelity; using it as the condition makes the diffusion model focus more on learning the video dynamics; and the video decoder is trained over unlabeled video data, thus benefiting from high-quality easily-available videos. VideoGen sets a new state-of-the-art in text-to-video generation in terms of both qualitative and quantitative evaluation. See urlhttps://videogen.github.io/VideoGen/ for more samples.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 60 of its 68 references that are in the map. First posted on arXiv 1 Sep 2023.

Builds on

60 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  4. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  6. 2022Video Diffusion Models7 Apr

Other references · 54

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2022Generative Adversarial Networkscited by 5,627
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
38 more
  1. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  8. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  10. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  11. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  13. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  14. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  15. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  16. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  17. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  18. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  19. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  20. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 635
  21. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 622
  22. 2021Designing a Practical Degradation Model for Deep Blind Image Super-Resolutioncited by 529
  23. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  24. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  25. 2022Novel View Synthesis with Diffusion Modelscited by 300
  26. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  27. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  28. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  29. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  30. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  31. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  32. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  33. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  34. 2017Video Generation From Textcited by 154
  35. 2022FILM: Frame Interpolation for Large Motioncited by 153
  36. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  37. 2022IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolationcited by 109
  38. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 89

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 18; 2024: 24; 2025: 22; 2026: 10.

74 papers.

Sort:

By area:

  1. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  2. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  3. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  4. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 251
  5. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  6. 2023A Survey on Video Diffusion Modelscited by 206
  7. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  8. 2023Panacea: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 168
  9. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  10. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  11. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  12. 2024Training-free Camera Control for Video Generationcited by 122
38 more
  1. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  2. 2024Loong: Generating Minute-level Long Videos with Autoregressive Language Modelscited by 99
  3. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 76
  4. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70
  5. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  6. 2024Towards A Better Metric for Text-to-Video Generationcited by 51
  7. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generationcited by 50
  8. 2023Breathing Life Into Sketches Using Text-to-Video Priorscited by 48
  9. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 42
  10. 2024OSV: One Step is Enough for High-Quality Image to Video Generationcited by 36
  11. 2024DreamForge: Motion-Aware Autoregressive Video Generation for Multi-View Driving Scenescited by 35
  12. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Modelscited by 30
  13. 2025CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Modelscited by 29
  14. 2024Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Modelcited by 28
  15. 2023MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generationcited by 26
  16. 2024MegaFusion: Extend Diffusion Models towards Higher-resolution Image Generation without Further Tuningcited by 25
  17. 2025VITA: Vision-to-Action Flow Matching Policycited by 25
  18. 2025Guiding Human-Object Interactions with Rich Geometry and Relationscited by 24
  19. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  20. 2025PresentAgent: Multimodal Agent for Presentation Video Generationcited by 21
  21. 2025Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generationcited by 20
  22. 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 19
  23. 2023MoVideo: Motion-Aware Video Generation with Diffusion Modelscited by 15
  24. 2024VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Modelscited by 14
  25. 2024I4VGen: Image as Free Stepping Stone for Text-to-Video Generationcited by 14
  26. 2025Presenting a Paper is an Art: Self-Improvement Aesthetic Agents for Academic Presentationscited by 13
  27. 2024MotiF: Making Text Count in Image Animation with Motion Focal Losscited by 10
  28. 2023FusionFrames: Efficient Architectural Aspects for Text-to-Video Generation Pipelinecited by 8
  29. 2023MotionZero:Exploiting Motion Priors for Zero-shot Text-to-Video Generationcited by 7
  30. 2024Spider: Any-to-Many Multimodal LLMcited by 6
  31. 2025UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?cited by 6
  32. 2025Learning Primitive Embodied World Models: Towards Scalable Robotic Learningcited by 6
  33. 2026Chain of Event-Centric Causal Thought for Physically Plausible Video Generationcited by 6
  34. 2024Panacea+: Panoramic and Controllable Video Generation for Autonomous Drivingcited by 5
  35. 2026From Ideal to Real: Stable Video Object Removal under Imperfect Conditionscited by 5
  36. 2024Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicalitycited by 4
  37. 2024ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipationcited by 4
  38. 2025HOFAR: High-Order Augmentation of Flow Autoregressive Transformerscited by 4

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  2. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation
  3. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
  4. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  5. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation
  6. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation
  7. 2023MEVG: Multi-event Video Generation with Text-to-Video Models
  8. 2024xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations