mlsanityData to 7 Oct 2026

Flexible Neural Representation for Physics Prediction

arXiv 1806.08047 ↗First posted 21 Jun 2018cs.AI cs.CV cs.LGcited by 91

Not on the map. Why?

Abstract (arXiv)

Humans have a remarkable capacity to understand the physical dynamics of objects in their environment, flexibly capturing complex structures and interactions at multiple levels of detail. Inspired by this ability, we propose a hierarchical particle-based object representation that covers a wide variety of types of three-dimensional objects, including both arbitrary rigid geometrical shapes and deformable materials. We then describe the Hierarchical Relation Network (HRN), an end-to-end differentiable neural network based on hierarchical graph convolution, that learns to predict physical dynamics in this representation. Compared to other neural network baselines, the HRN accurately handles complex collisions and nonrigid deformations, generating plausible dynamics predictions at long time scales in novel settings, and scaling to large scene configurations. These results demonstrate an architecture with the potential to form the basis of next-generation physics predictors for use in computer vision, robotics, and quantitative cognitive science.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 15; 2022: 29; 2023: 20; 2024: 14; 2025: 8; 2026: 4.

91 papers.

Sort:

By area:

  1. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 261
  2. 2021Towards Causal Representation Learningcited by 207
  3. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  4. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  5. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  6. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  7. 2021PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physicscited by 100
  8. 2023EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoningcited by 97
  9. 2022MultiScale MeshGraphNetscited by 94
  10. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
  11. 2022Equivariant Graph Mechanics Networks with Constraintscited by 67
  12. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
38 more
  1. 2022Geometrically Equivariant Graph Neural Networks: A Surveycited by 55
  2. 2021An End-to-End Differentiable Framework for Contact-Aware Robot Designcited by 53
  3. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 50
  4. 2020Deep Imitation Learning for Bimanual Robotic Manipulationcited by 49
  5. 2022DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Toolscited by 44
  6. 2022Learning rigid dynamics with face interaction graph networkscited by 43
  7. 2021Simulating Continuum Mechanics with Multi-Scale Graph Neural Networkscited by 39
  8. 2021Spline-PINN: Approaching PDEs without Data using Fast, Physics-Informed Hermite-Spline CNNscited by 38
  9. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 38
  10. 2022Learning Physical Dynamics with Subequivariant Graph Neural Networkscited by 38
  11. 2022Physical Design using Differentiable Learned Simulatorscited by 37
  12. 2024Equivariant Graph Neural Operator for Modeling 3D Dynamicscited by 37
  13. 2022NeuroFluid: Fluid Dynamics Grounding with Particle-Driven Neural Radiance Fieldscited by 36
  14. 2024A Survey of Geometric Graph Neural Networks: Data Structures, Models and Applicationscited by 28
  15. 2021Constraint-based graph network simulatorcited by 27
  16. 2023SEGNO: Generalizing Equivariant Graph Neural Networks with Physical Inductive Biasescited by 27
  17. 2023LagrangeBench: A Lagrangian Fluid Mechanics Benchmarking Suitecited by 26
  18. 2022Equivariant Graph Hierarchy-Based Neural Networkscited by 24
  19. 2022ACID: Action-Conditional Implicit Visual Dynamics for Deformable Object Manipulationcited by 24
  20. 2025Generative Physical AI in Vision: A Surveycited by 24
  21. 2023Dynamic-Resolution Model Learning for Object Pile Manipulationcited by 22
  22. 2023Deep Learning in Deterministic Computational Mechanicscited by 21
  23. 2022A Survey on Machine Learning Approaches for Modelling Intuitive Physicscited by 20
  24. 2025Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networkscited by 19
  25. 2021HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworkscited by 18
  26. 2023Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformercited by 18
  27. 2023Learning 3D Particle-based Simulators from RGB-D Videoscited by 17
  28. 2021SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learningcited by 16
  29. 2023Neural Relational Inference with Fast Modular Meta-learningcited by 16
  30. 2022Learning Large-scale Subsurface Simulations with a Hybrid Graph Network Simulatorcited by 15
  31. 2022Transformer with Implicit Edges for Particle-based Physics Simulationcited by 15
  32. 2023Grounding Graph Network Simulators using Physical Sensor Observationscited by 15
  33. 2022RelViT: Concept-guided Vision Transformer for Visual Relational Reasoningcited by 14
  34. 2021HighAir: A Hierarchical Graph Neural Network-Based Air Quality Forecasting Methodcited by 12
  35. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  36. 2022Towards Fast Simulation of Environmental Fluid Mechanics with Multi-Scale Graph Neural Networkscited by 11
  37. 2023Understanding Physical Dynamics with Counterfactual World Modelingcited by 11
  38. 2025Learning Coordinated Bimanual Manipulation Policies using State Diffusion and Inverse Dynamics Modelscited by 9

50 most cited of 91 All 91 (JSON)