# PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation

arXiv: https://arxiv.org/abs/2509.20358 · HTML: https://mlsanity.com/paper/2509.20358/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 76 of its 102 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 76)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)

## Built on by (most cited first, 30 of 41)

- [What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards](https://mlsanity.com/paper/2512.00425/) (2025)
- [Self-Refining Video Sampling](https://mlsanity.com/paper/2601.18577/) (2026)
- [Physical Simulator In-the-Loop Video Generation](https://mlsanity.com/paper/2603.06408/) (2026)
- [LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Model](https://mlsanity.com/paper/2603.27449/) (2026)
- [NEWTON: Agentic Planning for Physically Grounded Video Generation](https://mlsanity.com/paper/2605.18396/) (2026)
- [PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding](https://mlsanity.com/paper/2511.20562/) (2025)
- [VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System](https://mlsanity.com/paper/2607.27380/) (2026)
- [Planning with Sketch-Guided Verification for Physics-Aware Video Generation](https://mlsanity.com/paper/2511.17450/) (2025)
- [Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for now](https://mlsanity.com/paper/2512.02016/) (2025)
- [PhyCo: Learning Controllable Physical Priors for Generative Motion](https://mlsanity.com/paper/2604.28169/) (2026)
- [PhysAgent: Automating Physics-Based 4D Synthesis via Trajectory-Grounded Multi-Agent Feedback](https://mlsanity.com/paper/2606.08688/) (2026)
- [GS-Agent: Creating 4D Physical Worlds With Generative Simulation](https://mlsanity.com/paper/2607.21522/) (2026)
- [PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment](https://mlsanity.com/paper/2603.13770/) (2026)
- [Next-Scale Autoregressive Models for Text-to-Motion Generation](https://mlsanity.com/paper/2604.03799/) (2026)
- [PhysOmni: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction](https://mlsanity.com/paper/2605.20290/) (2026)
- [LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation](https://mlsanity.com/paper/2605.23878/) (2026)
- [WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformer](https://mlsanity.com/paper/2605.15305/) (2026)
- [Phys4D: Fine-Grained Physics-Consistent 4D Modeling from Video Diffusion](https://mlsanity.com/paper/2603.03485/) (2026)
- [Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows](https://mlsanity.com/paper/2603.21210/) (2026)
- [Lighting-grounded Video Generation with Renderer-based Agent Reasoning](https://mlsanity.com/paper/2604.07966/) (2026)
- [Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generation](https://mlsanity.com/paper/2606.25306/) (2026)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [PhysAgent: Reflective Agentic Physics Control for Physically Plausible Video Generation](https://mlsanity.com/paper/2607.16355/) (2026)
- [Principia: Relational Physics Tests for Video Models](https://mlsanity.com/paper/2609.04200/) (2026)
- [TourPhysics: Bringing Physics to World Models for Exploration and Manipulation from a Single Image](https://mlsanity.com/paper/2609.04911/) (2026)
- [Seeking Physics in Diffusion Noise](https://mlsanity.com/paper/2603.14294/) (2026)
- [UniPixie: Unified and Probabilistic 3D Physics Learning via Flow Matching](https://mlsanity.com/paper/2606.05399/) (2026)
- [Streaming Video Generation with Streaming Force Control](https://mlsanity.com/paper/2606.07508/) (2026)
- [NEXUS: Neural Energy Fields for Physically Consistent Contact-Rich 3D Object Dynamics](https://mlsanity.com/paper/2606.15015/) (2026)
- [A Study of the Design Space of Motion and Disocclusion Control in Video Generation](https://mlsanity.com/paper/2512.13392/) (2025)

## Abstract (arXiv)

Existing video generation models excel at producing photo-realistic videos from text or images, but often lack physical plausibility and 3D controllability. To overcome these limitations, we introduce PhysCtrl, a novel framework for physics-grounded image-to-video generation with physical parameters and force control. At its core is a generative physics network that learns the distribution of physical dynamics across four materials (elastic, sand, plasticine, and rigid) via a diffusion model conditioned on physics parameters and applied forces. We represent physical dynamics as 3D point trajectories and train on a large-scale synthetic dataset of 550K animations generated by physics simulators. We enhance the diffusion model with a novel spatiotemporal attention block that emulates particle interactions and incorporates physics-based constraints during training to enforce physical plausibility. Experiments show that PhysCtrl generates realistic, physics-grounded motion trajectories which, when used to drive image-to-video models, yield high-fidelity, controllable videos that outperform existing methods in both visual quality and physical plausibility. Project Page: https://cwchenwang.github.io/physctrl