mlsanityData to 14 Oct 2026

VideoPhy: Evaluating Physical Commonsense for Video Generation

arXiv 2406.03520 ↗First posted 5 Jun 2024cs.CV cs.AI cs.LGcited by 228

Not on the map. Why?

Cited by 58 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in internet-scale video data pretraining have led to the development of text-to-video generative models that can create high-quality videos across a broad range of visual concepts, synthesize realistic motions and render complex objects. Hence, these generative models have the potential to become general-purpose simulators of the physical world. However, it is unclear how far we are from this goal with the existing text-to-video generative models. To this end, we present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities (e.g. marbles will roll down when placed on a slanted surface). Specifically, we curate diverse prompts that involve interactions between various material types in the physical world (e.g., solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on these captions from diverse state-of-the-art text-to-video generative models, including open models (e.g., CogVideoX) and closed models (e.g., Lumiere, Dream Machine). Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts, while also lack physical commonsense. Specifically, the best performing model, CogVideoX-5B, generates videos that adhere to the caption and physical laws for 39.6% of the instances. VideoPhy thus highlights that the video generative models are far from accurately simulating the physical world. Finally, we propose an auto-evaluator, VideoCon-Physics, to assess the performance reliably for the newly released models.

Key facts. Cited by 228 papers in the map (data through 14 Oct 2026). Builds on 78 of its 118 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

78 of 118 references.

References · 78

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2022Scalable Diffusion Models with Transformerscited by 6,054
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
38 more
  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  2. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  8. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  9. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  12. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  13. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  14. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  15. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  16. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 938
  17. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  18. 2024Genie: Generative Interactive Environmentscited by 747
  19. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  20. 2023ModelScope Text-to-Video Technical Reportcited by 658
  21. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  22. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  23. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  24. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  25. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  26. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  27. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  28. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  29. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  30. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  31. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  32. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  33. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  34. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  35. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 377
  36. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  37. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  38. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 83; 2026: 133.

228 papers.

Sort:

By area:

  1. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  2. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  3. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  4. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  5. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  6. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 174
  7. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  8. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 149
  9. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 124
  10. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 98
  11. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 90
  12. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
38 more
  1. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 80
  2. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 62
  3. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  4. 2025IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environmentscited by 52
  5. 2025PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Dropcited by 50
  6. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  7. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 49
  8. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 48
  9. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  10. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  11. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  12. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 42
  13. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  14. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 41
  15. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 41
  16. 2025T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generationcited by 39
  17. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 39
  18. 2026WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluationcited by 39
  19. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
  20. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 37
  21. 2025"PhyWorldBench": A Comprehensive Evaluation of Physical Realism in Text-to-Video Modelscited by 36
  22. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Modelscited by 35
  23. 2025D3: Training-Free AI-Generated Video Detection Using Second-Order Featurescited by 34
  24. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  25. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  26. 2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 31
  27. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 30
  28. 2024STIV: Scalable Text and Image Conditioned Video Generationcited by 25
  29. 2025Generative Physical AI in Vision: A Surveycited by 24
  30. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  31. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 22
  32. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  33. 2025QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Searchcited by 20
  34. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 18
  35. 2026MBench: A Comprehensive Benchmark on Memory Capability for Video World Modelscited by 18
  36. 2025Learning Human-Perceived Fakeness in AI-Generated Videos via Multimodal LLMscited by 17
  37. 2025Impossible Videoscited by 16
  38. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusioncited by 16

50 most cited of 228 All 228 (JSON)