mlsanityData to 14 Oct 2026

Physion: Evaluating Physical Prediction from Vision in Humans and Machines

arXiv 2106.08261 ↗First posted 15 Jun 2021cs.AI cs.CVcited by 153

Not on the map. Why?

Cited by 34 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do.

Key facts. Cited by 153 papers in the map (data through 14 Oct 2026). Builds on 51 of its 74 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

51 of 74 references.

References · 51

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  4. 2015Fast R-CNNcited by 2,933
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  6. 2018Datasheets for Datasetscited by 1,386
  7. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  8. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  9. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  10. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  11. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  12. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
38 more
  1. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  2. 2018Stochastic Video Generation with a Learned Priorcited by 245
  3. 2018Graph networks as learnable physics engines for inference and controlcited by 244
  4. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 220
  5. 2019Contrastive Learning of Structured World Modelscited by 209
  6. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 204
  7. 2018Stochastic Adversarial Video Predictioncited by 173
  8. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
  9. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  10. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  11. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  12. 2018Flexible Neural Representation for Physics Predictioncited by 91
  13. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  14. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  15. 2018IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoningcited by 73
  16. 2016Learning Physical Intuition of Block Towers by Examplecited by 68
  17. 2019Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoningcited by 61
  18. 2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
  19. 2019COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Explorationcited by 57
  20. 2018ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stackingcited by 55
  21. 2020Visual Grounding of Learned Physical Modelscited by 55
  22. 2018Reasoning About Physical Interactions with Object-Oriented Prediction and Planningcited by 51
  23. 2020Learning Physical Graph Representations from Visual Scenescited by 50
  24. 2020Goal-Aware Prediction: Learning to Model What Matterscited by 45
  25. 2020Learning Long-term Visual Dynamics with Region Proposal Interaction Networkscited by 44
  26. 2019Compositional Video Predictioncited by 43
  27. 2019Learning Predictive Models From Observation and Interactioncited by 41
  28. 2018Learning to Play with Intrinsically-Motivated Self-Aware Agentscited by 35
  29. 2015Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Imagescited by 26
  30. 2016"What happens if..." Learning to Predict the Effect of Forces in Imagescited by 23
  31. 2018Relational Forward Models for Multi-Agent Learningcited by 21
  32. 2018Robustness via Retrying: Closed-Loop Robotic Manipulation with Self-Supervised Learningcited by 19
  33. 2019Combining Physical Simulators and Object-Based Networks for Controlcited by 18
  34. 2020Forward Prediction for Physical Reasoningcited by 18
  35. 2018Probing Physics Knowledge Using Tools from Developmental Psychologycited by 12
  36. 2018Modeling human intuitions about liquid flow with particle-based simulationcited by 12
  37. 2016To Fall Or Not To Fall: A Visual Approach to Physical Stability Predictioncited by 10
  38. 2016A Comparative Evaluation of Approximate Probabilistic Simulation and Deep Neural Networks as Accounts of Human Physical Scene Understandingcited by 9

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 5; 2023: 18; 2024: 21; 2025: 32; 2026: 72.

153 papers.

Sort:

By area:

  1. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  2. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  3. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modelingcited by 103
  4. 2025VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Modelscited by 98
  5. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  6. 20253D and 4D World Modeling: A Surveycited by 67
  7. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 54
  8. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  9. 2025IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environmentscited by 52
  10. 2025PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Dropcited by 50
  11. 2022Learning rigid dynamics with face interaction graph networkscited by 43
  12. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
38 more
  1. 2022Learning Physical Dynamics with Subequivariant Graph Neural Networkscited by 39
  2. 2024Core Knowledge Deficits in Multi-Modal Language Modelscited by 38
  3. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 37
  4. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 34
  5. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  6. 2024LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Enginescited by 30
  7. 2024Scaling 4D Representationscited by 30
  8. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  9. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 30
  10. 2024A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applicationscited by 28
  11. 2023A Control-Centric Benchmark for Video Predictioncited by 25
  12. 2025Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixelscited by 24
  13. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 22
  14. 2025Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulationscited by 21
  15. 2026Large Language Model Reasoning Failurescited by 21
  16. 2022A Survey on Machine Learning Approaches for Modelling Intuitive Physicscited by 20
  17. 2022Neural Implicit Representations for Physical Parameter Inference from a Single Videocited by 19
  18. 2023Unifying (Machine) Vision via Counterfactual World Modelingcited by 19
  19. 2024Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answeringcited by 18
  20. 2025LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preferencecited by 16
  21. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  22. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interactioncited by 15
  23. 2023SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstractioncited by 14
  24. 2023Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenescited by 13
  25. 2025Brain-Model Evaluations Need the NeuroAI Turing Testcited by 13
  26. 2026ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Modelcited by 13
  27. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  28. 2024Compositional Physical Reasoning of Objects and Events from Videoscited by 12
  29. 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Conceptscited by 12
  30. 2023Understanding Physical Dynamics with Counterfactual World Modelingcited by 11
  31. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Modelscited by 11
  32. 2025Learning Interactive World Model for Object-Centric Reinforcement Learningcited by 11
  33. 2021Phy-Q as a measure for physical reasoning intelligencecited by 10
  34. 2026PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinementcited by 10
  35. 2023Robo360: A 3D Omnispective Multi-Material Robotic Manipulation Datasetcited by 9
  36. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  37. 2026MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Modelscited by 9
  38. 2026WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesiscited by 9

50 most cited of 153 All 153 (JSON)