# Graph networks as learnable physics engines for inference and control

arXiv: https://arxiv.org/abs/1806.01242 · HTML: https://mlsanity.com/paper/1806.01242/

## Key facts

- Cited by 244 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2018.

## Built on by (most cited first, 30 of 244)

- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation](https://mlsanity.com/paper/2409.18964/) (2024)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [Learning Causally Invariant Representations for Out-of-Distribution Generalization on Graphs](https://mlsanity.com/paper/2202.05441/) (2022)
- [Pre-training via Denoising for Molecular Property Prediction](https://mlsanity.com/paper/2206.00133/) (2022)
- [Revisiting Over-smoothing and Over-squashing Using Ollivier-Ricci Curvature](https://mlsanity.com/paper/2211.15779/) (2022)
- [Learned Coarse Models for Efficient Turbulence Simulation](https://mlsanity.com/paper/2112.15275/) (2021)
- [MetaMorph: Learning Universal Controllers with Transformers](https://mlsanity.com/paper/2203.11931/) (2022)
- [Size-Invariant Graph Representations for Graph Classification Extrapolations](https://mlsanity.com/paper/2103.05045/) (2021)
- [Predicting Physics in Mesh-reduced Space with Temporal Attention](https://mlsanity.com/paper/2201.09113/) (2022)
- [AirfRANS: High Fidelity Computational Fluid Dynamics Dataset for Approximating Reynolds-Averaged Navier-Stokes Solutions](https://mlsanity.com/paper/2212.07564/) (2022)
- [A Survey on Interpretable Reinforcement Learning](https://mlsanity.com/paper/2112.13112/) (2021)
- [Port-Hamiltonian Neural Networks for Learning Explicit Time-Dependent Dynamical Systems](https://mlsanity.com/paper/2107.08024/) (2021)
- [When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning](https://mlsanity.com/paper/2203.16797/) (2022)
- [Learning Multi-Object Dynamics with Compositional Neural Radiance Fields](https://mlsanity.com/paper/2202.11855/) (2022)
- [A Generalist Neural Algorithmic Learner](https://mlsanity.com/paper/2209.11142/) (2022)
- [GraphFramEx: Towards Systematic Evaluation of Explainability Methods for Graph Neural Networks](https://mlsanity.com/paper/2206.09677/) (2022)
- [Multimodal learning with graphs](https://mlsanity.com/paper/2209.03299/) (2022)
- [Learning to schedule job-shop problems: Representation and policy learning using graph neural network and reinforcement learning](https://mlsanity.com/paper/2106.01086/) (2021)
- [Graph Inverse Reinforcement Learning from Diverse Videos](https://mlsanity.com/paper/2207.14299/) (2022)
- [Efficient Learning of Mesh-Based Physical Simulation with BSMS-GNN](https://mlsanity.com/paper/2210.02573/) (2022)
- [Simple GNN Regularisation for 3D Molecular Property Prediction & Beyond](https://mlsanity.com/paper/2106.07971/) (2021)
- [Physics-Guided Deep Learning for Dynamical Systems: A Survey](https://mlsanity.com/paper/2107.01272/) (2021)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [Learning rigid dynamics with face interaction graph networks](https://mlsanity.com/paper/2212.03574/) (2022)
- [Simulating Continuum Mechanics with Multi-Scale Graph Neural Networks](https://mlsanity.com/paper/2106.04900/) (2021)

## Abstract (arXiv)

Understanding and interacting with everyday physical scenes requires rich knowledge about the structure of the world, represented either implicitly in a value or policy function, or explicitly in a transition model. Here we introduce a new class of learnable models--based on graph networks--which implement an inductive bias for object- and relation-centric representations of complex, dynamical systems. Our results show that as a forward model, our approach supports accurate predictions from real and simulated data, and surprisingly strong and efficient generalization, across eight distinct physical systems which we varied parametrically and structurally. We also found that our inference model can perform system identification. Our models are also differentiable, and support online planning via gradient-based trajectory optimization, as well as offline policy optimization. Our framework offers new opportunities for harnessing and exploiting rich knowledge about the world, and takes a key step toward building machines with more human-like representations of the world.