# VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation

arXiv: https://arxiv.org/abs/2503.06800 · HTML: https://mlsanity.com/paper/2503.06800/

## Key facts

- Cited by 149 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 62 references that are in the map.
- First posted on arXiv 9 Mar 2025.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)
- [Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models](https://mlsanity.com/paper/2402.17177/) (2024)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://mlsanity.com/paper/2310.11440/) (2023)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation](https://mlsanity.com/paper/2406.15252/) (2024)
- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)
- [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://mlsanity.com/paper/2410.05363/) (2024)
- [Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generation](https://mlsanity.com/paper/2409.04410/) (2024)

## Built on by (most cited first, 30 of 149)

- [World Simulation with Video Foundation Models for Physical AI](https://mlsanity.com/paper/2511.00062/) (2025)
- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models](https://mlsanity.com/paper/2505.23656/) (2025)
- [WoW: Towards a World omniscient World model Through Embodied Interaction](https://mlsanity.com/paper/2509.22642/) (2025)
- [Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems](https://mlsanity.com/paper/2504.02918/) (2025)
- [PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation](https://mlsanity.com/paper/2509.20358/) (2025)
- [NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics](https://mlsanity.com/paper/2509.21309/) (2025)
- [VideoScore2: Think before You Score in Generative Video Evaluation](https://mlsanity.com/paper/2509.22799/) (2025)
- [WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation](https://mlsanity.com/paper/2605.25874/) (2026)
- [Inference-time Physics Alignment of Video Generative Models with Latent World Models](https://mlsanity.com/paper/2601.10553/) (2026)
- ["PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Models](https://mlsanity.com/paper/2507.13428/) (2025)
- [PAI-Bench: A Comprehensive Benchmark For Physical AI](https://mlsanity.com/paper/2512.01989/) (2025)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards](https://mlsanity.com/paper/2512.00425/) (2025)
- [PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation](https://mlsanity.com/paper/2512.24551/) (2025)
- [AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation](https://mlsanity.com/paper/2604.08540/) (2026)
- [QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models](https://mlsanity.com/paper/2512.19526/) (2025)
- [TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility](https://mlsanity.com/paper/2510.07550/) (2025)
- [PhysRVG: Physics-Aware Unified Reinforcement Learning for Video Generative Models](https://mlsanity.com/paper/2601.11087/) (2026)
- [Think Before You Diffuse: Infusing Physical Rules into Video Diffusion](https://mlsanity.com/paper/2505.21653/) (2025)
- [LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preference](https://mlsanity.com/paper/2510.11512/) (2025)
- [MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models](https://mlsanity.com/paper/2510.19022/) (2025)
- [Self-Refining Video Sampling](https://mlsanity.com/paper/2601.18577/) (2026)
- [Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoning](https://mlsanity.com/paper/2603.19607/) (2026)
- [Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibility](https://mlsanity.com/paper/2509.24702/) (2025)
- [Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Test](https://mlsanity.com/paper/2601.04137/) (2026)
- [Out of Sight, Out of Mind? Evaluating State Evolution in Video World Models](https://mlsanity.com/paper/2603.13215/) (2026)
- [WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts](https://mlsanity.com/paper/2601.21282/) (2026)
- [Phantom: Physics-Infused Video Generation via Joint Modeling of Visual and Latent Physical Dynamics](https://mlsanity.com/paper/2604.08503/) (2026)
- [PhyGround: Benchmarking Physical Reasoning in Generative World Models](https://mlsanity.com/paper/2605.10806/) (2026)

## Abstract (arXiv)

Large-scale video generative models, capable of creating realistic videos of diverse visual concepts, are strong candidates for general-purpose physical world simulators. However, their adherence to physical commonsense across real-world actions remains unclear (e.g., playing tennis, backflip). Existing benchmarks suffer from limitations such as limited size, lack of human evaluation, sim-to-real gaps, and absence of fine-grained physical rule analysis. To address this, we introduce VideoPhy-2, an action-centric dataset for evaluating physical commonsense in generated videos. We curate 200 diverse actions and detailed prompts for video synthesis from modern generative models. We perform human evaluation that assesses semantic adherence, physical commonsense, and grounding of physical rules in the generated videos. Our findings reveal major shortcomings, with even the best model achieving only 22% joint performance (i.e., high semantic and physical commonsense adherence) on the hard subset of VideoPhy-2. We find that the models particularly struggle with conservation laws like mass and momentum. Finally, we also train VideoPhy-AutoEval, an automatic evaluator for fast, reliable assessment on our dataset. Overall, VideoPhy-2 serves as a rigorous benchmark, exposing critical gaps in video generative models and guiding future research in physically-grounded video generation. The data and code is available at https://videophy2.github.io/.