# Think Before You Diffuse: Infusing Physical Rules into Video Diffusion

arXiv: https://arxiv.org/abs/2505.21653 · HTML: https://mlsanity.com/paper/2505.21653/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 66 references that are in the map.
- First posted on arXiv 27 May 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 32)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models](https://mlsanity.com/paper/2305.13655/) (2023)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity](https://mlsanity.com/paper/2502.01776/) (2025)
- [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024)
- [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://mlsanity.com/paper/2410.05363/) (2024)
- [Do generative video models understand physical principles?](https://mlsanity.com/paper/2501.09038/) (2025)
- [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023)
- [VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation](https://mlsanity.com/paper/2503.06800/) (2025)
- [VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning](https://mlsanity.com/paper/2309.15091/) (2023)
- [Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control](https://mlsanity.com/paper/2503.14492/) (2025)
- [One-Minute Video Generation with Test-Time Training](https://mlsanity.com/paper/2504.05298/) (2025)
- [PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation](https://mlsanity.com/paper/2412.00596/) (2024)
- [RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers](https://mlsanity.com/paper/2502.15894/) (2025)
- [DensePhysNet: Learning Dense Physical Object Representations via Multi-step Dynamic Interactions](https://mlsanity.com/paper/1906.03853/) (2019)
- [VidGen-1M: A Large-Scale Dataset for Text-to-video Generation](https://mlsanity.com/paper/2408.02629/) (2024)
- [Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Models](https://mlsanity.com/paper/2501.08453/) (2025)
- [FlashVideo: Flowing Fidelity to Detail for Efficient High-Resolution Video Generation](https://mlsanity.com/paper/2502.05179/) (2025)
- [Compositional 3D-aware Video Generation with LLM Director](https://mlsanity.com/paper/2409.00558/) (2024)

## Built on by (most cited first, 16 of 16)

- [Self-Refining Video Sampling](https://mlsanity.com/paper/2601.18577/) (2026)
- [VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System](https://mlsanity.com/paper/2607.27380/) (2026)
- [MoReGen: Multi-Agent Motion-Reasoning Engine for Code-based Text-to-Video Synthesis](https://mlsanity.com/paper/2512.04221/) (2025)
- [ReasonX: MLLM-Guided Intrinsic Image Decomposition](https://mlsanity.com/paper/2512.04222/) (2025)
- [Endless World: Real-Time 3D-Aware Long Video Generation](https://mlsanity.com/paper/2512.12430/) (2025)
- [Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement](https://mlsanity.com/paper/2511.20280/) (2025)
- [Physics in 2-Steps: Locking Motion Priors Before Visual Refinement Erases Them](https://mlsanity.com/paper/2606.06361/) (2026)
- [Phys4D: Fine-Grained Physics-Consistent 4D Modeling from Video Diffusion](https://mlsanity.com/paper/2603.03485/) (2026)
- [Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows](https://mlsanity.com/paper/2603.21210/) (2026)
- [Seeking Physics in Diffusion Noise](https://mlsanity.com/paper/2603.14294/) (2026)
- [Distilling Physical Priors into Streaming World Models](https://mlsanity.com/paper/2608.07981/) (2026)
- [PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.26916/) (2026)
- [VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement](https://mlsanity.com/paper/2609.03153/) (2026)
- [Physis-Lang: Self-Evolving Language as a Physical Representation for Video World Model](https://mlsanity.com/paper/2609.40358/) (2026)
- [Video Generation Models: A Survey of Post-Training and Alignment](https://mlsanity.com/paper/2610.00812/) (2026)
- [World Models' Last Exam in Physics](https://mlsanity.com/paper/2610.08791/) (2026)

## Abstract (arXiv)

Recent video diffusion models have demonstrated their great capability in generating visually-pleasing results, while synthesizing the correct physical effects in generated videos remains challenging. The complexity of real-world motions, interactions, and dynamics introduce great difficulties when learning physics from data. In this work, we propose DiffPhy, a generic framework that enables physically-correct and photo-realistic video generation by fine-tuning a pre-trained video diffusion model. Our method leverages large language models (LLMs) to infer rich physical context from the text prompt. To incorporate this context into the video diffusion model, we use a multimodal large language model (MLLM) to verify intermediate latent variables against the inferred physical rules, guiding the gradient updates of model accordingly. Textual output of LLM is transformed into continuous signals. We then formulate a set of training objectives that jointly ensure physical accuracy and semantic alignment with the input text. Additionally, failure facts of physical phenomena are corrected via attention injection. We also establish a high-quality physical video dataset containing diverse phyiscal actions and events to facilitate effective finetuning. Extensive experiments on public benchmarks demonstrate that DiffPhy is able to produce state-of-the-art results across diverse physics-related scenarios. Our project page is available at https://bwgzk-keke.github.io/DiffPhy/.