mlsanityData to 14 Oct 2026

LLM-grounded Video Diffusion Models

arXiv 2309.17444 ↗First posted 29 Sep 2023cs.CV cs.AI cs.CLcited by 83

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-conditioned diffusion models have emerged as a promising tool for neural video generation. However, current models still struggle with intricate spatiotemporal prompts and often generate restricted or incorrect motion. To address these limitations, we introduce LLM-grounded Video Diffusion (LVD). Instead of directly generating videos from the text inputs, LVD first leverages a large language model (LLM) to generate dynamic scene layouts based on the text inputs and subsequently uses the generated layouts to guide a diffusion model for video generation. We show that LLMs are able to understand complex spatiotemporal dynamics from text alone and generate layouts that align closely with both the prompts and the object motion patterns typically observed in the real world. We then propose to guide video diffusion models with these layouts by adjusting the attention maps. Our approach is training-free and can be integrated into any video diffusion model that admits classifier guidance. Our results demonstrate that LVD significantly outperforms its base video diffusion model and several strong baseline methods in faithfully generating videos with the desired attributes and motion patterns.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 46 of its 58 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

46 of 58 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023ModelScope Text-to-Video Technical Report12 Aug
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  4. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  5. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 40

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
28 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  6. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  7. 2021Grounded Language-Image Pre-trainingcited by 1,156
  8. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  9. 2022Diffusion Models in Vision: A Surveycited by 1,015
  10. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  11. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  12. 2016Simple Online and Realtime Trackingcited by 868
  13. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  14. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  15. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  16. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 313
  17. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  18. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
  19. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 256
  20. 2018Stochastic Video Generation with a Learned Priorcited by 245
  21. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  22. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  23. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 164
  24. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
  25. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  26. 2022Generating Long Videos of Dynamic Scenescited by 113
  27. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  28. 2022Shape-Guided Diffusion with Inside-Outside Attentioncited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 28; 2025: 27; 2026: 20.

83 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  2. 2024The Platonic Representation Hypothesiscited by 382
  3. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  4. 2023A Survey on Video Diffusion Modelscited by 206
  5. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  6. 2023Self-correcting LLM-controlled Diffusion Modelscited by 113
  7. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 80
  8. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  9. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 68
  10. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 66
  11. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  12. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
38 more
  1. 2024VideoTetris: Towards Compositional Text-to-Video Generationcited by 51
  2. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generationcited by 50
  3. 2026Infinite Worlds with Versatile Interactionscited by 48
  4. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 43
  5. 2024Mind the Time: Temporally-Controlled Multi-Event Video Generationcited by 40
  6. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 40
  7. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  8. 2024PosterLlama: Bridging Design Ability of Langauge Model to Contents-Aware Layout Generationcited by 35
  9. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 31
  10. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  11. 2024A Review of Multi-Modal Large Language and Vision Modelscited by 30
  12. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 27
  13. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 25
  14. 2025PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verificationcited by 22
  15. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 21
  16. 2024Compositional 3D-aware Video Generation with LLM Directorcited by 18
  17. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 18
  18. 2024GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaborationcited by 17
  19. 2024MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Modelscited by 17
  20. 2024SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generationcited by 15
  21. 2025BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representationscited by 15
  22. 2024MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with Mixture of Score Guidancecited by 14
  23. 2024SceneGPT: A Language Model for 3D Scene Understandingcited by 12
  24. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  25. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLMcited by 9
  26. 2025Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initializationcited by 9
  27. 2023MotionZero:Exploiting Motion Priors for Zero-shot Text-to-Video Generationcited by 7
  28. 2024DreamRunner: Fine-Grained Compositional Story-to-Video Generation with Retrieval-Augmented Motion Adaptationcited by 7
  29. 2024Video Is Worth a Thousand Images: Exploring the Latest Trends in Long Video Generationcited by 7
  30. 2026Evolution of Video Generative Foundationscited by 7
  31. 2025C-Drag: Chain-of-Thought Driven Motion Controller for Video Generationcited by 6
  32. 2025Planning with Sketch-Guided Verification for Physics-Aware Video Generationcited by 6
  33. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  34. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 6
  35. 2025TempoControl: Temporal Attention Guidance for Text-to-Video Modelscited by 5
  36. 2026ActionParty: Multi-Subject Action Binding in Generative Video Gamescited by 5
  37. 2026GroundShot: Visually Consistent Multi-Shot Long Video Generation via Entity-Grounded Shot Schedulingcited by 5
  38. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generation
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  3. 2024Video Diffusion Models: A Survey
  4. 2022Video Diffusion Models
  5. 2025EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation
  6. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applications
  7. 2023MeDM: Mediating Image Diffusion Models for Video-to-Video Translation with Temporal Correspondence Guidance
  8. 2023Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs