# Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers

arXiv: https://arxiv.org/abs/2209.11943 · HTML: https://mlsanity.com/paper/2209.11943/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Sep 2022.

## Built on by (most cited first, 12 of 12)

- [Vision-based Manipulation from Single Human Video with Open-World Object Graphs](https://mlsanity.com/paper/2405.20321/) (2024)
- [Composable Part-Based Manipulation](https://mlsanity.com/paper/2405.05876/) (2024)
- [Point Cloud Models Improve Visual Robustness in Robotic Learners](https://mlsanity.com/paper/2404.18926/) (2024)
- [SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions](https://mlsanity.com/paper/2410.18416/) (2024)
- [Structural Concept Learning via Graph Attention for Multi-Level Rearrangement Planning](https://mlsanity.com/paper/2309.02547/) (2023)
- [NOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors](https://mlsanity.com/paper/2311.01530/) (2023)
- [Out of Sight, Still in Mind: Reasoning and Planning about Unobserved Objects with Video Tracking Enabled Memory Models](https://mlsanity.com/paper/2309.15278/) (2023)
- [Graph learning in robotics: a survey](https://mlsanity.com/paper/2310.04294/) (2023)
- [Discovering Predictive Relational Object Symbols with Symbolic Attentive Layers](https://mlsanity.com/paper/2309.00889/) (2023)
- [Multi-Object Graph Affordance Network: Goal-Oriented Planning through Learned Compound Object Affordances](https://mlsanity.com/paper/2309.10426/) (2023)
- [A Modern Take on Visual Relationship Reasoning for Grasp Planning](https://mlsanity.com/paper/2409.02035/) (2024)
- [Robotic Test Tube Rearrangement Using Combined Reinforcement Learning and Motion Planning](https://mlsanity.com/paper/2401.09772/) (2024)

## Abstract (arXiv)

Objects rarely sit in isolation in human environments. As such, we'd like our robots to reason about how multiple objects relate to one another and how those relations may change as the robot interacts with the world. To this end, we propose a novel graph neural network framework for multi-object manipulation to predict how inter-object relations change given robot actions. Our model operates on partial-view point clouds and can reason about multiple objects dynamically interacting during the manipulation. By learning a dynamics model in a learned latent graph embedding space, our model enables multi-step planning to reach target goal relations. We show our model trained purely in simulation transfers well to the real world. Our planner enables the robot to rearrange a variable number of objects with a range of shapes and sizes using both push and pick and place skills.