mlsanityData to 7 Oct 2026

Learning Visual Predictive Models of Physics for Playing Billiards

arXiv 1511.07404 ↗First posted 23 Nov 2015cs.CVcited by 58

Not on the map. Why?

Abstract (arXiv)

The ability to plan and execute goal specific actions in varied, unexpected settings is a central requirement of intelligent agents. In this paper, we explore how an agent can be equipped with an internal model of the dynamics of the external world, and how it can use this model to plan novel actions by running multiple internal simulations ("visual imagination"). Our models directly process raw visual input, and use a novel object-centric prediction formulation based on visual glimpses centered on objects (fixations) to enforce translational invariance of the learned physical laws. The agent gathers training data through random interaction with a collection of different environments, and the resulting model can then be used to plan goal-directed actions in novel environments that the agent has not seen before. We demonstrate that our agent can accurately plan actions for playing a simulated billiards game, which requires pushing a ball into a target position or into collision with another ball.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 8; 2023: 5; 2024: 10; 2025: 11; 2026: 7.

58 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  3. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  4. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  5. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  6. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  7. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 53
  8. 2021Learning the Predictability of the Futurecited by 48
  9. 2024Physical Property Understanding from Language-Embedded Feature Fieldscited by 45
  10. 2021Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planningcited by 43
  11. 2024Neural Material Adaptor for Visual Grounding of Intrinsic Dynamicscited by 30
  12. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
38 more
  1. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 27
  2. 2021Block Contextual MDPs for Continual Learningcited by 25
  3. 2021SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulationcited by 24
  4. 2021CausalCity: Complex Simulations with Agency for Causal Discovery and Reasoningcited by 23
  5. 2021Efficient and Interpretable Robot Manipulation with Graph Neural Networkscited by 21
  6. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  7. 2022A Survey on Machine Learning Approaches for Modelling Intuitive Physicscited by 20
  8. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  9. 2021HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworkscited by 18
  10. 2024Learning Physics From Video: Unsupervised Physical Parameter Estimation for Continuous Dynamical Systemscited by 13
  11. 2025FreeGave: 3D Physics Learning from Dynamic Videos by Gaussian Velocitycited by 13
  12. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  13. 2023Understanding Physical Dynamics with Counterfactual World Modelingcited by 11
  14. 2021Discovering User-Interpretable Capabilities of Black-Box Planning Agentscited by 10
  15. 2022PulseImpute: A Novel Benchmark Task for Pulsative Physiological Signal Imputationcited by 9
  16. 2021PIP: Physical Interaction Prediction via Mental Simulation with Span Selectioncited by 8
  17. 2025Seeing the Wind from a Falling Leafcited by 7
  18. 2021Object and Relation Centric Representations for Push Effect Predictioncited by 5
  19. 2021Object-centric Video Prediction without Annotationcited by 5
  20. 2022Pretraining on Interactions for Learning Grounded Affordance Representationscited by 5
  21. 2025Inferring Dynamic Physical Properties from Video Foundation Modelscited by 5
  22. 2025Breaking the Discretization Barrier of Continuous Physics Simulation Learningcited by 4
  23. 2023Inferring Fluid Dynamics via Inverse Renderingcited by 3
  24. 2025CueTip: An Interactive and Explainable Physics-aware Pool Assistantcited by 3
  25. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  26. 2022Distilling Governing Laws and Source Input for Dynamical Systems from Videoscited by 2
  27. 2023A Critical View of Vision-Based Long-Term Dynamics Prediction Under Environment Misalignmentcited by 2
  28. 2024Machine Learning Aided Modeling of Granular Materials: A Reviewcited by 2
  29. 2026Envisioning the Future, One Step at a Timecited by 2
  30. 2021Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videoscited by 1
  31. 2022Recognising Affordances in Predicted Futures to Plan with Consideration of Non-canonical Affordance Effectscited by 1
  32. 2024Learning Physical Dynamics for Object-centric Visual Predictioncited by 1
  33. 2024Learning Disentangled Representation in Object-Centric Models for Visual Dynamics Prediction via Transformerscited by 1
  34. 2025PAVAS: Physics-Aware Video-to-Audio Synthesiscited by 1
  35. 2021Motion Prediction Performance Analysis for Autonomous Driving Systems and the Effects of Tracking Noisecited by 0
  36. 2021Intuitive Physics Guided Exploration for Sample Efficient Sim2real Transfercited by 0
  37. 2022Combining Commonsense Reasoning and Knowledge Acquisition to Guide Deep Learning in Roboticscited by 0
  38. 2022Learning to reason about and to act on physical cascading eventscited by 0

50 most cited of 58 All 58 (JSON)