# NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics

arXiv: https://arxiv.org/abs/2509.21309 · HTML: https://mlsanity.com/paper/2509.21309/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 80 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 30 of 66)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [How Far is Video Generation from World Model: A Physical Law Perspective](https://mlsanity.com/paper/2411.02385/) (2024)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)

## Built on by (most cited first, 30 of 41)

- [Inference-time Physics Alignment of Video Generative Models with Latent World Models](https://mlsanity.com/paper/2601.10553/) (2026)
- [What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards](https://mlsanity.com/paper/2512.00425/) (2025)
- [PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Models](https://mlsanity.com/paper/2601.11087/) (2026)
- [LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference](https://mlsanity.com/paper/2510.11512/) (2025)
- [RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligence](https://mlsanity.com/paper/2512.02622/) (2025)
- [NEWTON: Agentic Planning for Physically Grounded Video Generation](https://mlsanity.com/paper/2605.18396/) (2026)
- [ProPhy: Progressive Physical Alignment for Dynamic World Simulation](https://mlsanity.com/paper/2512.05564/) (2025)
- [ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models](https://mlsanity.com/paper/2605.08567/) (2026)
- [PhysInOne: Visual Physics Learning and Reasoning in One Suite](https://mlsanity.com/paper/2604.09415/) (2026)
- [The Pulse of Motion: Measuring Physical Frame Rate from Visual Dynamics](https://mlsanity.com/paper/2603.14375/) (2026)
- [PhyRPR: Training-Free Physics-Constrained Video Generation](https://mlsanity.com/paper/2601.09255/) (2026)
- [Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for now](https://mlsanity.com/paper/2512.02016/) (2025)
- [SeeU: Seeing the Unseen World via 4D Dynamics-aware Generation](https://mlsanity.com/paper/2512.03350/) (2025)
- [MoReGen: Multi-Agent Motion-Reasoning Engine for Code-based Text-to-Video Synthesis](https://mlsanity.com/paper/2512.04221/) (2025)
- [GS-Agent: Creating 4D Physical Worlds With Generative Simulation](https://mlsanity.com/paper/2607.21522/) (2026)
- [Counterfactual World Models via Digital Twin-conditioned Video Diffusion](https://mlsanity.com/paper/2511.17481/) (2025)
- [PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment](https://mlsanity.com/paper/2603.13770/) (2026)
- [LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation](https://mlsanity.com/paper/2605.23878/) (2026)
- [VERTIGO: Visual Preference Optimization for Cinematic Camera Trajectory Generation](https://mlsanity.com/paper/2604.02467/) (2026)
- [Phys4D: Fine-Grained Physics-Consistent 4D Modeling from Video Diffusion](https://mlsanity.com/paper/2603.03485/) (2026)
- [OptiWorld: Optimal Control for Video World Generation under Physical Constraints](https://mlsanity.com/paper/2606.00499/) (2026)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [Principia: Relational Physics Tests for Video Models](https://mlsanity.com/paper/2609.04200/) (2026)
- [Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms](https://mlsanity.com/paper/2609.23658/) (2026)
- [FlowSteer: Conditioning Flow Field for Consistent Image Restoration](https://mlsanity.com/paper/2512.08125/) (2025)
- [Seeking Physics in Diffusion Noise](https://mlsanity.com/paper/2603.14294/) (2026)
- [LaWM: Least Action World Models for Long-Horizon Physical Consistency from Visual Observations](https://mlsanity.com/paper/2605.08279/) (2026)
- [NewtPhys: Do Foundation Models Understand Newtonian Physics?](https://mlsanity.com/paper/2606.03986/) (2026)
- [Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment](https://mlsanity.com/paper/2606.04737/) (2026)
- [NEXUS: Neural Energy Fields for Physically Consistent Contact-Rich 3D Object Dynamics](https://mlsanity.com/paper/2606.15015/) (2026)

## Abstract (arXiv)

A primary bottleneck in large-scale text-to-video generation today is physical consistency and controllability. Despite recent advances, state-of-the-art models often produce unrealistic motions, such as objects falling upward, or abrupt changes in velocity and direction. Moreover, these models lack precise parameter control, struggling to generate physically consistent dynamics under different initial conditions. We argue that this fundamental limitation stems from current models learning motion distributions solely from appearance, while lacking an understanding of the underlying dynamics. In this work, we propose NewtonGen, a framework that integrates data-driven synthesis with learnable physical principles. At its core lies trainable Neural Newtonian Dynamics (NND), which can model and predict a variety of Newtonian motions, thereby injecting latent dynamical constraints into the video generation process. By jointly leveraging data priors and dynamical guidance, NewtonGen enables physically consistent video synthesis with precise parameter control. All data and code are available at https://github.com/pandayuanyu/NewtonGen