mlsanityData to 7 Oct 2026

VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning

arXiv 2309.15091 ↗First posted 26 Sep 2023cs.CV cs.AI cs.CLcited by 138

Not on the map. Why?

Cited by 32 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent text-to-video (T2V) generation methods have seen significant advancements. However, the majority of these works focus on producing short video clips of a single event (i.e., single-scene videos). Meanwhile, recent large language models (LLMs) have demonstrated their capability in generating layouts and programs to control downstream visual modules. This prompts an important question: can we leverage the knowledge embedded in these LLMs for temporally consistent long video generation? In this paper, we propose VideoDirectorGPT, a novel framework for consistent multi-scene video generation that uses the knowledge of LLMs for video content planning and grounded video generation. Specifically, given a single text prompt, we first ask our video planner LLM (GPT-4) to expand it into a 'video plan', which includes the scene descriptions, the entities with their respective layouts, the background for each scene, and consistency groupings of the entities. Next, guided by this video plan, our video generator, named Layout2Vid, has explicit control over spatial layouts and can maintain temporal consistency of entities across multiple scenes, while being trained only with image-level annotations. Our experiments demonstrate that our proposed VideoDirectorGPT framework substantially improves layout and movement control in both single- and multi-scene video generation and can generate multi-scene videos with consistency, while achieving competitive performance with SOTAs in open-domain single-scene T2V generation. Detailed ablation studies, including dynamic adjustment of layout control strength with an LLM and video generation with user-provided images, confirm the effectiveness of each component of our framework and its future potential.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Builds on 53 of its 66 references that are in the map. First posted on arXiv 26 Sep 2023.

Builds on

53 of 66 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2020Denoising Diffusion Implicit Modelscited by 9,655
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2013Auto-Encoding Variational Bayescited by 3,980
38 more
  1. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  10. 2017Dense-Captioning Events in Videoscited by 789
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  12. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  13. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  14. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  15. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  16. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  17. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  18. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  19. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  20. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  21. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  22. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  23. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  24. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  25. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  26. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  27. 2017Video Generation From Textcited by 153
  28. 2018StoryGAN: A Sequential Conditional GAN for Story Visualizationcited by 137
  29. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  30. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  31. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  32. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  33. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  34. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  35. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  36. 2022NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesiscited by 71
  37. 2017DeepStory: Video Story QA by Deep Embedded Memory Networkscited by 67
  38. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 66

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 32; 2025: 44; 2026: 50.

138 papers.

Sort:

By area:

  1. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  2. 2023A Survey on Video Diffusion Modelscited by 203
  3. 2023Video Understanding with Large Language Models: A Surveycited by 203
  4. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  5. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  6. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  7. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  8. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  9. 2023LLM-grounded Video Diffusion Modelscited by 83
  10. 2025PlanGenLLMs: A Modern Survey of LLM Planning Capabilitiescited by 82
  11. 2025Automated Movie Generation via Multi-Agent CoT Planningcited by 82
  12. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
38 more
  1. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  2. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjectscited by 76
  3. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 64
  4. 2025RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformerscited by 63
  5. 2024VideoStudio: Generating Consistent-Content and Multi-Scene Videoscited by 62
  6. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  7. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 58
  8. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  9. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  10. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  11. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 42
  12. 2024Efficient Video Diffusion Models via Content-Frame Motion-Latent Decompositioncited by 41
  13. 2024A Survey on Long Video Generation: Challenges, Methods, and Prospectscited by 40
  14. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 39
  15. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  16. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  17. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  18. 2023DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planningcited by 28
  19. 2025FilMaster: Bridging Cinematic Principles and Generative AI for Automated Film Generationcited by 28
  20. 2023Plan, Posture and Go: Towards Open-World Text-to-Motion Generationcited by 26
  21. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  22. 2024TALC: Time-Aligned Captions for Multi-Scene Text-to-Video Generationcited by 25
  23. 2025Mask2DiT: Dual Mask-based Diffusion Transformer for Multi-Scene Long Video Generationcited by 24
  24. 2026VideoMemory: Toward Consistent Video Generation via Memory Integrationcited by 22
  25. 2025T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generationcited by 19
  26. 2024Compositional 3D-aware Video Generation with LLM Directorcited by 18
  27. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 17
  28. 2025Generative AI for Cel-Animation: A Surveycited by 17
  29. 2025AniMaker: Multi-Agent Animated Storytelling with MCTS-Driven Clip Generationcited by 17
  30. 2024MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Modelscited by 16
  31. 2025Test-time Prompt Refinement for Text-to-Image Modelscited by 16
  32. 2025BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representationscited by 15
  33. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusioncited by 14
  34. 2024MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptionscited by 13
  35. 2024MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidancecited by 13
  36. 2025Eliciting In-context Retrieval and Reasoning for Long-context Large Language Modelscited by 11
  37. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  38. 2026ViMax: Agentic Video Generationcited by 11

50 most cited of 138 All 138 (JSON)