# Unsupervised Segmentation in Real-World Images via Spelke Object Inference

arXiv: https://arxiv.org/abs/2205.08515 · HTML: https://mlsanity.com/paper/2205.08515/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 17 May 2022.

## Builds on (most cited first, 28 of 28)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets](https://mlsanity.com/paper/2109.13396/) (2021)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://mlsanity.com/paper/1902.09843/) (2019)
- [ParseNet: Looking Wider to See Better](https://mlsanity.com/paper/1506.04579/) (2015)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [SSAP: Single-Shot Instance Segmentation With Affinity Pyramid](https://mlsanity.com/paper/1909.01616/) (2019)
- [SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition](https://mlsanity.com/paper/2106.03849/) (2021)
- [Learning Physical Graph Representations from Visual Scenes](https://mlsanity.com/paper/2006.12373/) (2020)
- [Unsupervised Object Learning via Common Fate](https://mlsanity.com/paper/2110.06562/) (2021)
- [Unsupervised Discovery of 3D Physical Objects from Video](https://mlsanity.com/paper/2007.12348/) (2020)
- [Unsupervised part representation by Flow Capsules](https://mlsanity.com/paper/2011.13920/) (2020)
- [MVTec D2S: Densely Segmented Supermarket Dataset](https://mlsanity.com/paper/1804.08292/) (2018)
- [A topological solution to object segmentation and tracking](https://mlsanity.com/paper/2107.02036/) (2021)

## Built on by (most cited first, 15 of 15)

- [Provably Learning Object-Centric Representations](https://mlsanity.com/paper/2305.14229/) (2023)
- [Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns](https://mlsanity.com/paper/2210.12148/) (2022)
- [Unifying (Machine) Vision via Counterfactual World Modeling](https://mlsanity.com/paper/2306.01828/) (2023)
- [Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images](https://mlsanity.com/paper/2210.02324/) (2022)
- [Unsupervised Discovery of Object-Centric Neural Fields](https://mlsanity.com/paper/2402.07376/) (2024)
- [Understanding Physical Dynamics with Counterfactual World Modeling](https://mlsanity.com/paper/2312.06721/) (2023)
- [Transformer brain encoders explain human high-level visual responses](https://mlsanity.com/paper/2505.17329/) (2025)
- [Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans](https://mlsanity.com/paper/2306.00294/) (2023)
- [Benchmarking and Analysis of Unsupervised Object Segmentation from Real-world Single Images](https://mlsanity.com/paper/2312.04947/) (2023)
- [Robot Instance Segmentation with Few Annotations for Grasping](https://mlsanity.com/paper/2407.01302/) (2024)
- [Measuring uncertainty in human visual segmentation](https://mlsanity.com/paper/2301.07807/) (2023)
- [Self-Supervised Instance Segmentation by Grasping](https://mlsanity.com/paper/2305.06305/) (2023)
- [The Background Also Matters: Background-Aware Motion-Guided Objects Discovery](https://mlsanity.com/paper/2311.02633/) (2023)
- [Human-like Object Grouping in Self-supervised Vision Transformers](https://mlsanity.com/paper/2603.13994/) (2026)
- [Improving action classification with brain-inspired deep networks](https://mlsanity.com/paper/2512.07729/) (2025)

## Abstract (arXiv)

Self-supervised, category-agnostic segmentation of real-world images is a challenging open problem in computer vision. Here, we show how to learn static grouping priors from motion self-supervision by building on the cognitive science concept of a Spelke Object: a set of physical stuff that moves together. We introduce the Excitatory-Inhibitory Segment Extraction Network (EISEN), which learns to extract pairwise affinity graphs for static scenes from motion-based training signals. EISEN then produces segments from affinities using a novel graph propagation and competition network. During training, objects that undergo correlated motion (such as robot arms and the objects they move) are decoupled by a bootstrapping process: EISEN explains away the motion of objects it has already learned to segment. We show that EISEN achieves a substantial improvement in the state of the art for self-supervised image segmentation on challenging synthetic and real-world robotics datasets.