mlsanityData to 14 Oct 2026

Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation

arXiv 2410.05363 ↗First posted 7 Oct 2024cs.CVcited by 174

Not on the map. Why?

Cited by 48 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text-to-video (T2V) models like Sora have made significant strides in visualizing complex prompts, which is increasingly viewed as a promising path towards constructing the universal world simulator. Cognitive psychologists believe that the foundation for achieving this goal is the ability to understand intuitive physics. However, the capacity of these models to accurately represent intuitive physics remains largely unexplored. To bridge this gap, we introduce PhyGenBench, a comprehensive Physics Generation Benchmark designed to evaluate physical commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully crafted prompts across 27 distinct physical laws, spanning four fundamental domains, which could comprehensively assesses models' understanding of physical commonsense. Alongside PhyGenBench, we propose a novel evaluation framework called PhyGenEval. This framework employs a hierarchical evaluation structure utilizing appropriate advanced vision-language models and large language models to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can conduct large-scale automated assessments of T2V models' understanding of physical commonsense, which align closely with human feedback. Our evaluation results and in-depth analysis demonstrate that current models struggle to generate videos that comply with physical commonsense. Moreover, simply scaling up models or employing prompt engineering techniques is insufficient to fully address the challenges presented by PhyGenBench (e.g., dynamic scenarios). We hope this study will inspire the community to prioritize the learning of physical commonsense in these models beyond entertainment applications. We will release the data and codes at https://github.com/OpenGVLab/PhyGenBench

Key facts. Cited by 174 papers in the map (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

29 of 38 references.

References · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  7. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  8. 2017The Kinetics Human Action Video Datasetcited by 1,909
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  10. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  11. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  12. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
17 more
  1. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  3. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  4. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  5. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  6. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  7. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  8. 2022Generating Long Videos of Dynamic Scenescited by 113
  9. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  10. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  11. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 75
  12. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 67
  13. 2024VEnhancer: Generative Space-Time Enhancement for Video Generationcited by 64
  14. 2024VidGen-1M: A Large-Scale Dataset for Text-to-video Generationcited by 62
  15. 2024Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?cited by 52
  16. 2023GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language modelscited by 28
  17. 2024Object permanence in newborn chicks is robust against opposing evidencecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 55; 2026: 113.

174 papers.

Sort:

By area:

  1. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  2. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  3. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  4. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 149
  5. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 137
  6. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 98
  7. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 90
  8. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 80
  9. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  10. 2025PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Dropcited by 50
  11. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  12. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 48
38 more
  1. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  2. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 42
  3. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 41
  4. 2025RoboScape: Physics-informed Embodied World Modelcited by 40
  5. 2025T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generationcited by 39
  6. 2026WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluationcited by 39
  7. 2024PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulationcited by 36
  8. 2025"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Modelscited by 36
  9. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Modelscited by 35
  10. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  11. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 34
  12. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  13. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  14. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 30
  15. 2025HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generationcited by 26
  16. 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Modelscited by 26
  17. 2025Generative Physical AI in Vision: A Surveycited by 24
  18. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  19. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 22
  20. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  21. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generationcited by 19
  22. 2025TiViBench: Benchmarking Think-in-Video Reasoning for Video Generative Modelscited by 19
  23. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibilitycited by 18
  24. 2026MBench: A Comprehensive Benchmark on Memory Capability for Video World Modelscited by 18
  25. 2025Impossible Videoscited by 16
  26. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  27. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusioncited by 16
  28. 2025LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preferencecited by 16
  29. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  30. 2025Science-T2I: Addressing Scientific Illusions in Image Synthesiscited by 15
  31. 2025PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selectioncited by 15
  32. 2026Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoningcited by 14
  33. 2025Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibilitycited by 13
  34. 2025RULER-Bench: Probing Rule-based Reasoning Abilities of Next-level Video Generation Models for Vision Foundation Intelligencecited by 13
  35. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Testcited by 13
  36. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 13
  37. 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Conceptscited by 12
  38. 2026VQQA: An Agentic Approach for Video Evaluation and Quality Improvementcited by 12

50 most cited of 174 All 174 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025PhysVidBench: Language-Grounded Evaluation of Physical Commonsense in Text-to-Video Models
  2. 2026Physis-Lang: Self-Evolving Language as a Physical Representation for Video World Model
  3. 2025Can World Simulators Reason? Gen-ViRe: A Generative Visual Reasoning Benchmark
  4. 2025Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement
  5. 2025MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation
  6. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
  7. 2026PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation
  8. 2025A Physical Coherence Benchmark for Evaluating Video Generation Models via Optical Flow-guided Frame Prediction