mlsanityData to 7 Oct 2026

A Compositional Object-Based Approach to Learning Physical Dynamics

arXiv 1612.00341 ↗First posted 1 Dec 2016cs.AI cs.LGcited by 119

Not on the map. Why?

Abstract (arXiv)

We present the Neural Physics Engine (NPE), a framework for learning simulators of intuitive physics that naturally generalize across variable object count and different scene configurations. We propose a factorization of a physical scene into composable object-based representations and a neural network architecture whose compositional structure factorizes object dynamics into pairwise interactions. Like a symbolic physics engine, the NPE is endowed with generic notions of objects and their interactions; realized as a neural network, it can be trained via stochastic gradient descent to adapt to specific object properties and dynamics of different worlds. We evaluate the efficacy of our approach on simple rigid body dynamics in two-dimensional worlds. By comparing to less structured architectures, we show that the NPE's compositional representation of the structure in physical interactions improves its ability to predict movement, generalize across variable object count and different scene configurations, and infer latent properties of objects such as mass.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 25; 2023: 23; 2024: 19; 2025: 8; 2026: 12.

119 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  3. 2021Towards Causal Representation Learningcited by 207
  4. 2025Do generative video models understand physical principles?cited by 151
  5. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  6. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  7. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  8. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  9. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  10. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  11. 2022When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learningcited by 64
  12. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
38 more
  1. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 61
  2. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  3. 2021An End-to-End Differentiable Framework for Contact-Aware Robot Designcited by 53
  4. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 50
  5. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  6. 2022DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Toolscited by 44
  7. 2021Learning Neuro-Symbolic Relational Transition Models for Bilevel Planningcited by 43
  8. 2021Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planningcited by 43
  9. 2021Spatio-Temporal Dynamic Inference Network for Group Activity Recognitioncited by 42
  10. 2021Simulating Continuum Mechanics with Multi-Scale Graph Neural Networkscited by 39
  11. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 38
  12. 2025MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulationcited by 34
  13. 2021Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Modelscited by 33
  14. 2021Constructing Neural Network-Based Models for Simulating Dynamical Systemscited by 29
  15. 2021Conditional Neural Relational Inference for Interacting Systemscited by 28
  16. 2021Noether Networks: Meta-Learning Useful Conserved Quantitiescited by 26
  17. 2023Generalizing Graph ODE for Learning Complex System Dynamics across Environmentscited by 23
  18. 2023Dynamic-Resolution Model Learning for Object Pile Manipulationcited by 22
  19. 2024Understanding Video Transformers via Universal Concept Discoverycited by 22
  20. 2022Learning What and Where: Disentangling Location and Identity Tracking Without Supervisioncited by 21
  21. 2023Deep Learning in Deterministic Computational Mechanicscited by 21
  22. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  23. 2022A Survey on Machine Learning Approaches for Modelling Intuitive Physicscited by 20
  24. 2022Learning Efficient Abstract Planning Models that Choose What to Predictcited by 17
  25. 2023Learning Vortex Dynamics for Fluid Inference and Predictioncited by 17
  26. 2021SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learningcited by 16
  27. 2023Neural Relational Inference with Fast Modular Meta-learningcited by 16
  28. 2021RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networkscited by 15
  29. 2021Pushing it out of the Way: Interactive Visual Navigationcited by 14
  30. 2021Reasoning-Modulated Representationscited by 13
  31. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  32. 2021Active Learning of Abstract Plan Feasibilitycited by 11
  33. 2022Towards Fast Simulation of Environmental Fluid Mechanics with Multi-Scale Graph Neural Networkscited by 11
  34. 2024Compositional Physical Reasoning of Objects and Events from Videoscited by 11
  35. 2022Knowledge-augmented Deep Learning and Its Applications: A Surveycited by 10
  36. 2024ElastoGen: 4D Generative Elastodynamicscited by 10
  37. 2021Neural Graph Matching based Collaborative Filteringcited by 9
  38. 2022Motif-based Graph Representation Learning with Application to Chemical Moleculescited by 9

50 most cited of 119 All 119 (JSON)