mlsanityData to 7 Oct 2026

PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation

arXiv 2412.00596 ↗First posted 30 Nov 2024cs.CV cs.AIcited by 77

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-to-video (T2V) generation has been recently enabled by transformer-based diffusion models, but current T2V models lack capabilities in adhering to the real-world common knowledge and physical rules, due to their limited understanding of physical realism and deficiency in temporal modeling. Existing solutions are either data-driven or require extra model inputs, but cannot be generalizable to out-of-distribution domains. In this paper, we present PhyT2V, a new data-independent T2V technique that expands the current T2V model's capability of video generation to out-of-distribution domains, by enabling chain-of-thought and step-back reasoning in T2V prompting. Our experiments show that PhyT2V improves existing T2V models' adherence to real-world physical rules by 2.3x, and achieves 35% improvement compared to T2V prompt enhancers. The source codes are available at: https://github.com/pittisl/PhyT2V.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 34 of its 63 references that are in the map. First posted on arXiv 30 Nov 2024.

Builds on

34 of 63 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Compositional 3D-aware Video Generation with LLM Director31 Aug
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  3. 2024WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs10 Mar
  4. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjects18 Jan
  5. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  6. 2023Photorealistic Video Generation with Diffusion Models11 Dec
  7. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning21 Nov
  8. 2023LLM-grounded Video Diffusion Models29 Sep
  9. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep

Other references · 25

  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  5. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  6. 2023Learning Interactive Real-World Simulatorscited by 440
  7. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  8. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  9. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  10. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  11. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  12. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
13 more
  1. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  2. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  3. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  4. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  5. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  6. 2023Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answeringcited by 97
  7. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 87
  8. 2016UnrealCV: Connecting Computer Vision to Unreal Enginecited by 41
  9. 2024A Survey on Long Video Generation: Challenges, Methods, and Prospectscited by 40
  10. 2023Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagationcited by 12
  11. 2024Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitatorscited by 9
  12. 2024FreezeAsGuard: Mitigating Illegal Adaptation of Diffusion Models via Selective Tensor Freezingcited by 4
  13. 2024Game Engines for Immersive Visualization: Using Unreal Engine Beyond Entertainmentcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 33; 2026: 44.

77 papers.

Sort:

By area:

  1. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 88
  2. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 87
  3. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  4. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  5. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 40
  6. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  7. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  8. 2025Generative Physical AI in Vision: A Surveycited by 24
  9. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 23
  10. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 22
  11. 2025PresentAgent: Multimodal Agent for Presentation Video Generationcited by 20
  12. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 20
38 more
  1. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  2. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generationcited by 18
  3. 2025TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Modelscited by 18
  4. 2026AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generationcited by 18
  5. 2025Reasoning Physical Video Generation with Diffusion Timestep Tokens via Reinforcement Learningcited by 17
  6. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  7. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  8. 2026PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Modelscited by 15
  9. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusioncited by 14
  10. 2025Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibilitycited by 13
  11. 2025MoAlign: Motion-Centric Representation Alignment for Video Diffusion Modelscited by 13
  12. 2025RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligencecited by 12
  13. 2026Physical Simulator In-the-Loop Video Generationcited by 12
  14. 2025Grounding Creativity in Physics: A Brief Survey of Physical Priors in AIGCcited by 11
  15. 2025Presenting a Paper is an Art: Self-Improvement Aesthetic Agents for Academic Presentationscited by 11
  16. 2025ProPhy: Progressive Physical Alignment for Dynamic World Simulationcited by 11
  17. 2025Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Modelscited by 10
  18. 2025PhysMaster: Mastering Physical Representation for Video Generation via Reinforcement Learningcited by 10
  19. 2026Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamicscited by 10
  20. 2026NEWTON: Agentic Planning for Physically Grounded Video Generationcited by 10
  21. 2025RDPO: Real Data Preference Optimization for Physics Consistency Video Generationcited by 8
  22. 2025MoGAN: Improving Motion Quality in Video Diffusion via Few-Step Motion Adversarial Post-Trainingcited by 7
  23. 2025Planning with Sketch-Guided Verification for Physics-Aware Video Generationcited by 6
  24. 2026Chain of Event-Centric Causal Thought for Physically Plausible Video Generationcited by 6
  25. 2026Evolution of Video Generative Foundationscited by 6
  26. 2026Physics-IQ Verifiedcited by 6
  27. 2026PhysVid: Physics Aware Local Conditioning for Generative Video Modelscited by 5
  28. 2025PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Modelscited by 4
  29. 2026PhyPrompt: RL-based Prompt Refinement for Physically Plausible Text-to-Video Generationcited by 4
  30. 2026SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generationcited by 4
  31. 2026PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedbackcited by 4
  32. 2025SeeU: Seeing the Unseen World via 4D Dynamics-aware Generationcited by 3
  33. 2026PhyEduVideo: A Benchmark for Evaluating Text-to-Video Models for Physics Educationcited by 3
  34. 2026PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignmentcited by 3
  35. 2026PresentAgent-2: Towards Generalist Multimodal Presentation Agentscited by 3
  36. 2026When Physical Preferences Meet Semantic Constraints: Physical and Semantic Direct Preference Optimization for Text-to-Video Generationcited by 3
  37. 2026MMPhysVideo: Physically Plausible Video Generation Through Joint RGB-Perception Modelingcited by 2
  38. 2026Physics in 2-Steps: Locking Motion Priors Before Visual Refinement Erases Themcited by 2

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement
  2. 2026PhyRPR: Training-Free Physics-Constrained Video Generation
  3. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design
  4. 2025VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos?
  5. 2025MoReGen: Multi-Agent Motion-Reasoning Engine for Code-based Text-to-Video Synthesis
  6. 2025PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selection
  7. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models
  8. 2024Searching Priors Makes Text-to-Video Synthesis Better