mlsanityData to 14 Oct 2026

Reasoning About Physical Interactions with Object-Oriented Prediction and Planning

arXiv 1812.10972 ↗First posted 28 Dec 2018cs.LG cs.AI cs.CVcited by 51

Not on the map. Why?

Abstract (arXiv)

Object-based factorizations provide a useful level of abstraction for interacting with the world. Building explicit object representations, however, often requires supervisory signals that are difficult to obtain in practice. We present a paradigm for learning object-centric representations for physical scene understanding without direct supervision of object properties. Our model, Object-Oriented Prediction and Planning (O2P2), jointly learns a perception function to map from image observations to object representations, a pairwise physics interaction function to predict the time evolution of a collection of objects, and a rendering function to map objects back to pixels. For evaluation, we consider not only the accuracy of the physical predictions of the model, but also its utility for downstream tasks that require an actionable representation of intuitive physics. After training our model on an image prediction task, we can use its learned representations to build block towers more complicated than those observed during training.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 16; 2022: 14; 2023: 11; 2024: 8; 2025: 2.

51 papers.

Sort:

By area:

  1. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  2. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 153
  3. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  4. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  5. 2021AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactionscited by 57
  6. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  7. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
  8. 2021O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learningcited by 46
  9. 2021From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligencecited by 35
  10. 2022Compositional Scene Representation Learning via Reconstruction: A Surveycited by 27
  11. 2023A Survey on Physics Informed Reinforcement Learning: Review and Open Problemscited by 24
  12. 2021Structured World Belief for Reinforcement Learning in POMDPcited by 21
38 more
  1. 2022Learning What and Where: Disentangling Location and Identity Tracking Without Supervisioncited by 21
  2. 2022Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motionscited by 21
  3. 2022A Survey on Machine Learning Approaches for Modelling Intuitive Physicscited by 20
  4. 2023Systematic Visual Reasoning through Object-Centric Relational Abstractioncited by 20
  5. 2022Compositional Multi-Object Reinforcement Learning with Linear Relation Networkscited by 19
  6. 2023Learning 3D Particle-based Simulators from RGB-D Videoscited by 17
  7. 2021Pushing it out of the Way: Interactive Visual Navigationcited by 16
  8. 2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 13
  9. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  10. 2022On the Learning Mechanisms in Physical Reasoningcited by 11
  11. 2024PhyGrasp: Generalizing Robotic Grasping with Physics-informed Large Multimodal Modelscited by 11
  12. 2021Object-Centric Representation Learning with Generative Spatial-Temporal Factorizationcited by 10
  13. 2022Binding Actions to Objects in World Modelscited by 10
  14. 2021IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenescited by 9
  15. 2023Multi-Object Manipulation via Object-Centric Neural Scattering Functionscited by 9
  16. 2023Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generationcited by 7
  17. 2024FORCE: Physics-aware Human-object Interactioncited by 7
  18. 2024Reasoning-Enhanced Object-Centric Learning for Videoscited by 7
  19. 2022Factored World Models for Zero-Shot Generalization in Robotic Manipulationcited by 6
  20. 2022Graph-based Reinforcement Learning meets Mixed Integer Programs: An application to 3D robot assembly discoverycited by 6
  21. 2023Conceptual Framework for Autonomous Cognitive Entitiescited by 6
  22. 2021Object-centric Video Prediction without Annotationcited by 5
  23. 2023Perceiving Unseen 3D Objects by Poking the Objectscited by 5
  24. 2024Scaling Face Interaction Graph Networks to Real World Scenescited by 5
  25. 2025SmallWorlds: Assessing Dynamics Understanding of World Models in Isolated Environmentscited by 4
  26. 2021The Impact of Negative Sampling on Contrastive Structured World Modelscited by 2
  27. 2022Fixing Malfunctional Objects With Learned Physical Simulation and Functional Predictioncited by 2
  28. 2022Stable Object Reorientation using Contact Plane Registrationcited by 2
  29. 20236-DoF Stability Field via Diffusion Modelscited by 2
  30. 2021Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videoscited by 1
  31. 2022T3VIP: Transformation-based 3D Video Predictioncited by 1
  32. 2023Planning for Learning Object Propertiescited by 1
  33. 2024Learning Physical Dynamics for Object-centric Visual Predictioncited by 1
  34. 2024Guiding Video Prediction with Explicit Procedural Knowledgecited by 1
  35. 2021Intuitive Physics Guided Exploration for Sample Efficient Sim2real Transfercited by 0
  36. 2022Learning to reason about and to act on physical cascading eventscited by 0
  37. 2022Relate to Predict: Towards Task-Independent Knowledge Representations for Reinforcement Learningcited by 0
  38. 2024Interpretable Representation Learning from Videos using Nonlinear Priorscited by 0

50 most cited of 51 All 51 (JSON)