mlsanityData to 14 Oct 2026

GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning

arXiv 2311.12631 ↗First posted 21 Nov 2023cs.CVcited by 61

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advances in text-to-video generation have harnessed the power of diffusion models to create visually compelling content conditioned on text prompts. However, they usually encounter high computational costs and often struggle to produce videos with coherent physical motions. To tackle these issues, we propose GPT4Motion, a training-free framework that leverages the planning capability of large language models such as GPT, the physical simulation strength of Blender, and the excellent image generation ability of text-to-image diffusion models to enhance the quality of video synthesis. Specifically, GPT4Motion employs GPT-4 to generate a Blender script based on a user textual prompt, which commands Blender's built-in physics engine to craft fundamental scene components that encapsulate coherent physical motions across frames. Then these components are inputted into Stable Diffusion to generate a video aligned with the textual prompt. Experimental results on three basic physical motion scenarios, including rigid object drop and collision, cloth draping and swinging, and liquid flow, demonstrate that GPT4Motion can generate high-quality videos efficiently in maintaining motion coherency and entity consistency. GPT4Motion offers new insights in text-to-video research, enhancing its quality and broadening its horizon for further explorations.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 64 of its 73 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

64 of 73 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023LLM-grounded Video Diffusion Models29 Sep
  2. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  3. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  4. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation1 Sep
  5. 2023ModelScope Text-to-Video Technical Report12 Aug
  6. 2023DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis7 Aug
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  8. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  9. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  10. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  11. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  12. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
5 more
  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  3. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  5. 2022Video Diffusion Models7 Apr

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2022Training language models to follow instructions with human feedbackcited by 15,971
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
35 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2022Generative Adversarial Networkscited by 5,627
  4. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  13. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  14. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  16. 2023PaLM 2 Technical Reportcited by 1,162
  17. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  18. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  19. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  20. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  21. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  22. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  23. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  24. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 243
  25. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  26. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  27. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  28. 2023AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolationcited by 180
  29. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 164
  30. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 147
  31. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  32. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  33. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 49
  34. 2020Audio Captioning using Pre-Trained Large-Scale Language Model Guided by Audio-based Similar Caption Retrievalcited by 27
  35. 2023A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 24; 2026: 22.

61 papers.

Sort:

By area:

  1. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  2. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  3. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 134
  4. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 80
  5. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  6. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  7. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 41
  8. 2025RoboScape: Physics-informed Embodied World Modelcited by 40
  9. 2025T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generationcited by 39
  10. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  11. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  12. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 27
38 more
  1. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  2. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 25
  3. 2025Generative Physical AI in Vision: A Surveycited by 24
  4. 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understandingcited by 22
  5. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 21
  6. 2025LL3M: Large Language 3D Modelerscited by 20
  7. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 18
  8. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  9. 2026PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Modelscited by 17
  10. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  11. 2026Self-Refining Video Samplingcited by 15
  12. 2026Physical Simulator In-the-Loop Video Generationcited by 14
  13. 2025PhysMaster: Mastering Physical Representation for Video Generation via Reinforcement Learningcited by 12
  14. 2025Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Modelscited by 10
  15. 2025Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxycited by 9
  16. 2024Dual-Schedule Inversion: Training- and Tuning-Free Inversion for Real Image Editingcited by 8
  17. 2024Reconstructing Animals and the Wildcited by 7
  18. 2026VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstructioncited by 7
  19. 2024Llama Learns to Direct: DirectorLLM for Human-Centric Video Generationcited by 6
  20. 2026Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generationcited by 6
  21. 2025GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Designcited by 5
  22. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 5
  23. 2026BrickNet: Graph-Backed Generative Brick Assemblycited by 5
  24. 2026GS-Agent: Creating 4D Physical Worlds With Generative Simulationcited by 5
  25. 2025Multi-Object Sketch Animation by Scene Decomposition and Motion Planningcited by 4
  26. 2026EZBlender: Efficient 3D Editing with Plan-and-ReAct Agentcited by 4
  27. 2026PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignmentcited by 4
  28. 2024Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioningcited by 3
  29. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  30. 2025PhysTalk: Language-driven Real-time Physics in 3D Gaussian Scenescited by 3
  31. 2026DiReCT: Disentangled Regularization of Contrastive Trajectories for Physics-Refined Video Generationcited by 2
  32. 2026MoRight: Motion Control Done Rightcited by 2
  33. 2026Authoring for Living Worlds: Tool-Constrained LLM Agents for Executable Multi-Actor Scenarioscited by 2
  34. 2026Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanismscited by 2
  35. 2024InteractPro: A Unified Framework for Motion-Aware Image Compositioncited by 1
  36. 2025Learning to Animate Images from A Few Videos to Portray Delicate Human Actionscited by 1
  37. 2025PhysFire-WM: A Physics-Informed World Model for Emulating Fire Spread Dynamicscited by 1
  38. 2026Physics-Informed Video Diffusion For Shallow Water Equationscited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Make Pixels Dance: High-Dynamic Video Generation
  2. 2026From Diffusion to Flow: Efficient Motion Generation in MotionGPT3
  3. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
  4. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agent
  5. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models
  6. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior
  7. 2026CausalMotion: Structured Physical Reasoning as Keyframe and Trajectory Guidance for Training-Free Video Generation
  8. 2024Animate Your Motion: Turning Still Images into Dynamic Videos