# SlotGNN: Unsupervised Discovery of Multi-Object Representations and Visual Dynamics

arXiv: https://arxiv.org/abs/2310.04617 · HTML: https://mlsanity.com/paper/2310.04617/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 6 Oct 2023.

## Builds on (most cited first, 21 of 21)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/1802.04687/) (2018)
- [Deep Visual Foresight for Planning Robot Motion](https://mlsanity.com/paper/1610.00696/) (2016)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [Learning to Poke by Poking: Experiential Learning of Intuitive Physics](https://mlsanity.com/paper/1606.07419/) (2016)
- [Unsupervised Learning of Object Keypoints for Perception and Control](https://mlsanity.com/paper/1906.11883/) (2019)
- [Visual Interaction Networks](https://mlsanity.com/paper/1706.01433/) (2017)
- [Unsupervised Learning of Object Structure and Dynamics from Videos](https://mlsanity.com/paper/1906.07889/) (2019)
- [Propagation Networks for Model-Based Control Under Partial Observation](https://mlsanity.com/paper/1809.11169/) (2018)
- [Learning Multi-Object Dynamics with Compositional Neural Radiance Fields](https://mlsanity.com/paper/2202.11855/) (2022)
- [SORNet: Spatial Object-Centric Representations for Sequential Manipulation](https://mlsanity.com/paper/2109.03891/) (2021)
- [Learning Long-term Visual Dynamics with Region Proposal Interaction Networks](https://mlsanity.com/paper/2008.02265/) (2020)
- [Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations](https://mlsanity.com/paper/2205.06333/) (2022)
- [Object-centric Forward Modeling for Model Predictive Control](https://mlsanity.com/paper/1910.03568/) (2019)
- [KINet: Unsupervised Forward Models for Robotic Pushing Manipulation](https://mlsanity.com/paper/2202.09006/) (2022)

## Built on by (most cited first, 2 of 2)

- [Rethinking Object-Centric Representations for Video Dynamics Modeling](https://mlsanity.com/paper/2606.23436/) (2026)
- [CGSA: Class-Guided Slot-Aware Adaptation for Source-Free Object Detection](https://mlsanity.com/paper/2602.22621/) (2026)

## Abstract (arXiv)

Learning multi-object dynamics from visual data using unsupervised techniques is challenging due to the need for robust, object representations that can be learned through robot interactions. This paper presents a novel framework with two new architectures: SlotTransport for discovering object representations from RGB images and SlotGNN for predicting their collective dynamics from RGB images and robot interactions. Our SlotTransport architecture is based on slot attention for unsupervised object discovery and uses a feature transport mechanism to maintain temporal alignment in object-centric representations. This enables the discovery of slots that consistently reflect the composition of multi-object scenes. These slots robustly bind to distinct objects, even under heavy occlusion or absence. Our SlotGNN, a novel unsupervised graph-based dynamics model, predicts the future state of multi-object scenes. SlotGNN learns a graph representation of the scene using the discovered slots from SlotTransport and performs relational and spatial reasoning to predict the future appearance of each slot conditioned on robot actions. We demonstrate the effectiveness of SlotTransport in learning object-centric features that accurately encode both visual and positional information. Further, we highlight the accuracy of SlotGNN in downstream robotic tasks, including challenging multi-object rearrangement and long-horizon prediction. Finally, our unsupervised approach proves effective in the real world. With only minimal additional data, our framework robustly predicts slots and their corresponding dynamics in real-world control tasks.