mlsanityData to 7 Oct 2026

ComPhy: Compositional Physical Reasoning of Objects and Events from Videos

arXiv 2205.01089 ↗First posted 2 May 2022cs.CV cs.AI cs.LGcited by 61

Not on the map. Why?

Abstract (arXiv)

Objects' motions in nature are governed by complex interactions and their properties. While some properties, such as shape and material, can be identified via the object's visual appearances, others like mass and electric charge are not directly visible. The compositionality between the visible and hidden properties poses unique challenges for AI models to reason from the physical world, whereas humans can effortlessly infer them with limited observations. Existing studies on video reasoning mainly focus on visually observable elements such as object appearance, movement, and contact interaction. In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset. For a given set of objects, ComPhy includes few videos of them moving and interacting under different initial conditions. The model is evaluated based on its capability to unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions posted on one of the videos. Evaluation results of several state-of-the-art video reasoning models on ComPhy show unsatisfactory performance as they fail to capture these hidden properties. We further propose an oracle neural-symbolic framework named Compositional Physics Learner (CPL), combining visual perception, physical property learning, dynamic prediction, and symbolic execution into a unified framework. CPL can effectively identify objects' physical properties from their interactions and predict their dynamics to answer questions.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 2 May 2022.

Builds on

41 of 50 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Mask R-CNNcited by 7,259
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  7. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2016Matching Networks for One Shot Learningcited by 2,246
  10. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  11. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  12. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
29 more
  1. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  2. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  3. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  4. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  5. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  6. 2018Neural Relational Inference for Interacting Systemscited by 339
  7. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  8. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  9. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
  10. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  11. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  12. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  13. 2019PHYRE: A New Benchmark for Physical Reasoningcited by 137
  14. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  15. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  16. 2016A Compositional Object-Based Approach to Learning Physical Dynamicscited by 119
  17. 2019CoPhy: Counterfactual Learning of Physical Dynamicscited by 80
  18. 2018IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoningcited by 72
  19. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  20. 2016Learning Physical Intuition of Block Towers by Examplecited by 66
  21. 2018Propagation Networks for Model-Based Control Under Partial Observationcited by 64
  22. 2019Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Videocited by 60
  23. 2020Attention over learned object embeddings enables complex visual reasoningcited by 59
  24. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  25. 2020CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractionscited by 44
  26. 2020Visual Concept-Metaconcept Learningcited by 39
  27. 2021PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoningcited by 34
  28. 2020ESPRIT: Explaining Solutions to Physical Reasoning Taskscited by 16
  29. 2020A Competence-aware Curriculum for Visual Concepts Learning via Question Answeringcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 14; 2024: 14; 2025: 12; 2026: 16.

61 papers.

Sort:

By area:

  1. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  2. 2025PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videoscited by 78
  3. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  4. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  5. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 53
  6. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 49
  7. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  8. 2024Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modelingcited by 42
  9. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  10. 2022Learning Physical Dynamics with Subequivariant Graph Neural Networkscited by 38
  11. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Modelscited by 35
  12. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
38 more
  1. 2024LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Enginescited by 29
  2. 2023GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEscited by 18
  3. 2024Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answeringcited by 18
  4. 2023Advancing Visual Grounding with Scene Knowledge: Benchmark and Methodcited by 17
  5. 2023I-PHYRE: Interactive Physical Reasoningcited by 15
  6. 2022CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answeringcited by 13
  7. 2022On the Learning Mechanisms in Physical Reasoningcited by 11
  8. 2023CLEVRER-Humans: Describing Physical and Causal Events the Human Waycited by 11
  9. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Modelscited by 11
  10. 2024Compositional Physical Reasoning of Objects and Events from Videoscited by 11
  11. 2023ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Modelscited by 10
  12. 2023Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Followingcited by 9
  13. 2025We'll Fix it in Post: Improving Text-to-Video Generation with Neuro-Symbolic Feedbackcited by 9
  14. 2025PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesiscited by 9
  15. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  16. 2023Intrinsic Physical Concepts Discovery with Object-Centric Predictive Modelscited by 8
  17. 2023Benchmarks for Physical Reasoning AIcited by 8
  18. 2024Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Videocited by 8
  19. 2026MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Modelscited by 8
  20. 2022Neuro-Symbolic Learning: Principles and Applications in Ophthalmologycited by 7
  21. 2024Neural-Symbolic VideoQA: Learning Compositional Spatio-Temporal Reasoning for Real-world Video Question Answeringcited by 6
  22. 2024Interactive Learning of Physical Object Properties Through Robot Manipulation and Database of Object Measurementscited by 5
  23. 2024MARPLE: A Benchmark for Long-Horizon Inferencecited by 5
  24. 2024Integrating Physics and Topology in Neural Networks for Learning Rigid Body Dynamicscited by 5
  25. 2025A Challenge to Build Neuro-Symbolic Video Agentscited by 5
  26. 2026PhyCo: Learning Controllable Physical Priors for Generative Motioncited by 5
  27. 2023ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Modelscited by 4
  28. 2025NeuSpring: Neural Spring Fields for Reconstruction and Simulation of Deformable Objects from Videoscited by 4
  29. 2024One-shot Video Imitation via Parameterized Symbolic Abstraction Graphscited by 3
  30. 2025Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoningcited by 3
  31. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  32. 2026Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMscited by 3
  33. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  34. 2025ObjectAlign: Neuro-Symbolic Object Consistency Verification and Correctioncited by 2
  35. 2026PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Modelscited by 2
  36. 2026Learning Physics-Guided Residual Dynamics for Deformable Object Simulationcited by 2
  37. 2026PhiZero: A World Model Built Around Physical Languagecited by 2
  38. 2023Does Visual Pretraining Help End-to-End Reasoning?cited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction
  2. 2026PhysicsMind: Sim and Real Mechanics Benchmarking for Physical Reasoning and Prediction in Foundational VLMs and World Models
  3. 2024ContPhy: Continuum Physical Concept Learning and Reasoning from Videos
  4. 2025TRAVL: A Recipe for Making Video-Language Models Better Judges of Physics Implausibility
  5. 2026Asking the World: Generalist Physical Reasoning through Agentic World Modeling and Probing
  6. 2025QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models
  7. 2026PhysMRV: Physical Memory Retrieval and Verification for Physics Plausibility Reasoning
  8. 2026World Models' Last Exam in Physics