mlsanityData to 14 Oct 2026

CoPhy: Counterfactual Learning of Physical Dynamics

arXiv 1909.12000 ↗First posted 26 Sep 2019cs.CVcited by 81

Not on the map. Why?

Abstract (arXiv)

Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world. This work poses a new problem of counterfactual learning of object mechanics from visual input. We develop the CoPhy benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting. Having observed a mechanical experiment that involves, for example, a falling tower of blocks, a set of bouncing balls or colliding objects, we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions, such as displacing one of the objects in the scene. The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model in an end-to-end fashion with no supervision. We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment, which results in significantly more accurate predictions at the level of super human performance.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 9; 2023: 16; 2024: 10; 2025: 9; 2026: 22.

81 papers.

Sort:

By area:

  1. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 261
  2. 2025Do generative video models understand physical principles?cited by 161
  3. 2022Cross-view Geo-localization via Learning Disentangled Geometric Layout Correspondencecited by 70
  4. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  5. 2022When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learningcited by 65
  6. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 63
  7. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  8. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 54
  9. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  10. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  11. 2022A Review of Emerging Research Directions in Abstract Visual Reasoningcited by 33
  12. 2021ACRE: Abstract Causal REasoning Beyond Covariationcited by 30
38 more
  1. 2024LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Enginescited by 30
  2. 2023GeoDTR+: Toward generic cross-view geolocalization via geometric disentanglementcited by 28
  3. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 24
  4. 2023Generalizing Graph ODE for Learning Complex System Dynamics across Environmentscited by 23
  5. 2023Counterfactual Learning on Graphs: A Surveycited by 22
  6. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 22
  7. 2021Capturing the objects of vision with neural networkscited by 21
  8. 2022A Survey on Machine Learning Approaches for Modelling Intuitive Physicscited by 20
  9. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  10. 2022Variational Causal Dynamics: Discovering Modular World Models from Interventionscited by 16
  11. 2022Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Spacecited by 14
  12. 2023MVTrans: Multi-View Perception of Transparent Objectscited by 14
  13. 2022CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answeringcited by 13
  14. 2021Hopper: Multi-hop Transformer for Spatiotemporal Reasoningcited by 12
  15. 2024Compositional Physical Reasoning of Objects and Events from Videoscited by 12
  16. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12
  17. 2023CLEVRER-Humans: Describing Physical and Causal Events the Human Waycited by 11
  18. 2021Deep KKL: Data-driven Output Prediction for Non-Linear Systemscited by 10
  19. 2021Phy-Q as a measure for physical reasoning intelligencecited by 10
  20. 2025Towards Visual Discrimination and Reasoning of Real-World Physical Dynamics: Physics-Grounded Anomaly Detectioncited by 10
  21. 2021SPACE: A Simulator for Physical Interactions and Causal Learning in 3D Environmentscited by 9
  22. 2024Language Agents Meet Causality -- Bridging LLMs and Causal World Modelscited by 9
  23. 2025Learning Coordinated Bimanual Manipulation Policies using State Diffusion and Inverse Dynamics Modelscited by 9
  24. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  25. 2026Physics-IQ Verifiedcited by 9
  26. 2021PIP: Physical Interaction Prediction via Mental Simulation with Span Selectioncited by 8
  27. 2023Intrinsic Physical Concepts Discovery with Object-Centric Predictive Modelscited by 8
  28. 2023ACQUIRED: A Dataset for Answering Counterfactual Questions In Real-Life Videoscited by 8
  29. 2023Benchmarks for Physical Reasoning AIcited by 8
  30. 2026VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstructioncited by 7
  31. 2023A Survey of Methods, Challenges and Perspectives in Causalitycited by 6
  32. 2023Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Modelscited by 6
  33. 2025Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learningcited by 6
  34. 2022Causal Counterfactuals for Improving the Robustness of Reinforcement Learningcited by 5
  35. 2026PhyCo: Learning Controllable Physical Priors for Generative Motioncited by 5
  36. 2026TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraintcited by 5
  37. 2021OPEn: An Open-ended Physics Environment for Learning Without a Taskcited by 4
  38. 2024Automated Discovery of Functional Actual Causes in Complex Environmentscited by 4

50 most cited of 81 All 81 (JSON)