mlsanityData to 14 Oct 2026

How Far is Video Generation from World Model: A Physical Law Perspective

arXiv 2411.02385 ↗First posted 4 Nov 2024cs.CV cs.AIcited by 212

On the map: World models → Latent predictive world models

Abstract (arXiv)

OpenAI's Sora highlights the potential of video generation for developing world models that adhere to fundamental physical laws. However, the ability of video generation models to discover such laws purely from visual data without human priors can be questioned. A world model learning the true law should give predictions robust to nuances and correctly extrapolate on unseen scenarios. In this work, we evaluate across three key scenarios: in-distribution, out-of-distribution, and combinatorial generalization. We developed a 2D simulation testbed for object movement and collisions to generate videos deterministically governed by one or more classical mechanics laws. This provides an unlimited supply of data for large-scale experimentation and enables quantitative evaluation of whether the generated videos adhere to physical laws. We trained diffusion-based video generation models to predict object movements based on initial frames. Our scaling experiments show perfect generalization within the distribution, measurable scaling behavior for combinatorial generalization, but failure in out-of-distribution scenarios. Further experiments reveal two key insights about the generalization mechanisms of these models: (1) the models fail to abstract general physical rules and instead exhibit "case-based" generalization behavior, i.e., mimicking the closest training example; (2) when generalizing to new cases, models are observed to prioritize different factors when referencing training data: color > size > velocity > shape. Our study suggests that scaling alone is insufficient for video generation models to uncover fundamental physical laws, despite its role in Sora's broader success. See our project page at https://phyworld.github.io

Key facts. Cited by 212 papers in the map (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 64 of its 80 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

64 of 80 references.

References · 64

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
38 more
  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2022Video Diffusion Modelscited by 2,305
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  8. 2023Mastering Diverse Domains through World Modelscited by 1,253
  9. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  10. 2020Mastering Atari with Discrete World Modelscited by 1,007
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  13. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  14. 2024Genie: Generative Interactive Environmentscited by 747
  15. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  16. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  17. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  18. 2017Video Enhancement with Task-Oriented Flowcited by 619
  19. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  20. 2023Learning Interactive Real-World Simulatorscited by 458
  21. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
  22. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  23. 2022MAGVIT: Masked Generative Video Transformercited by 346
  24. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  25. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 327
  26. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 310
  27. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  28. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  29. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  30. 2024GenAD: Generative End-to-End Autonomous Drivingcited by 225
  31. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 222
  32. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  33. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 154
  34. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  35. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  36. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  37. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 104
  38. 2014Material Recognition in the Wild with the Materials in Context Databasecited by 79

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 97; 2026: 107.

212 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  2. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  3. 2025Video models are zero-shot learners and reasonerscited by 220
  4. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 179
  5. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  6. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  7. 2025Do generative video models understand physical principles?cited by 161
  8. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  9. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 124
  10. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 90
  11. 20253D and 4D World Modeling: A Surveycited by 67
  12. 2025Long-Context State-Space Video World Modelscited by 64
38 more
  1. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 58
  2. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  3. 2025End-to-End Training for Autoregressive Video Diffusion via Self-Resamplingcited by 52
  4. 2025PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Dropcited by 50
  5. 2025Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systemscited by 48
  6. 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
  7. 2025ReSim: Reliable World Simulation for Autonomous Drivingcited by 44
  8. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 43
  9. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulationcited by 42
  10. 2025NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamicscited by 41
  11. 2025RoboScape: Physics-informed Embodied World Modelcited by 40
  12. 2026Infinite-World: Scaling Interactive World Models to 1000-Frame Horizons via Pose-Free Hierarchical Memorycited by 40
  13. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 37
  14. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 35
  15. 2025AI-Generated Video Detection via Perceptual Straighteningcited by 34
  16. 2026A Very Big Video Reasoning Suitecited by 34
  17. 2025Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoningcited by 33
  18. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  19. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  20. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  21. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 30
  22. 2025Taming Teacher Forcing for Masked Autoregressive Video Generationcited by 30
  23. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 30
  24. 2026ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignmentcited by 29
  25. 2025GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulationcited by 27
  26. 2024Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesiscited by 26
  27. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 25
  28. 2025Generative Physical AI in Vision: A Surveycited by 24
  29. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 24
  30. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 23
  31. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  32. 2025FlowMo: Variance-Based Flow Guidance for Coherent Motion in Video Generationcited by 22
  33. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  34. 2025CheXWorld: Exploring Image World Modeling for Radiograph Representation Learningcited by 19
  35. 2025Reasoning Physical Video Generation with Diffusion Timestep Tokens via Reinforcement Learningcited by 18
  36. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 18
  37. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  38. 2025Think Before You Diffuse: Infusing Physical Rules into Video Diffusioncited by 16

50 most cited of 212 All 212 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PhyGround: Benchmarking Physical Reasoning in Generative World Models
  2. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithms
  3. 2026World Models' Last Exam in Physics
  4. 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts
  5. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals
  6. 2026Foresight at the Event Boundary: Evaluating Physical Prediction in Video World Models
  7. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
  8. 2026Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goals