# Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns

arXiv: https://arxiv.org/abs/2210.12148 · HTML: https://mlsanity.com/paper/2210.12148/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 77 references that are in the map.
- First posted on arXiv 21 Oct 2022.

## Builds on (most cited first, 30 of 49)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks](https://mlsanity.com/paper/2001.06810/) (2020)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)
- [ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/2111.10265/) (2021)
- [FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos](https://mlsanity.com/paper/1701.05384/) (2017)
- [SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping](https://mlsanity.com/paper/2105.07014/) (2021)
- [Unsupervised Moving Object Detection via Contextual Information Separation](https://mlsanity.com/paper/1901.03360/) (2019)
- [Improving Generative Imagination in Object-Centric World Models](https://mlsanity.com/paper/2010.02054/) (2020)
- [Tagger: Deep Unsupervised Perceptual Grouping](https://mlsanity.com/paper/1606.06724/) (2016)
- [SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition](https://mlsanity.com/paper/2106.03849/) (2021)

## Built on by (most cited first, 19 of 19)

- [Diffusion Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2306.09316/) (2023)
- [Self-supervised Object-Centric Learning for Videos](https://mlsanity.com/paper/2310.06907/) (2023)
- [Object Discovery from Motion-Guided Tokens](https://mlsanity.com/paper/2303.15555/) (2023)
- [Multi-Object Discovery by Low-Dimensional Object Motion](https://mlsanity.com/paper/2307.08027/) (2023)
- [Unsupervised Object Localization in the Era of Self-Supervised ViTs: A Survey](https://mlsanity.com/paper/2310.12904/) (2023)
- [Learning segmentation from point trajectories](https://mlsanity.com/paper/2501.12392/) (2025)
- [Are We Done with Object-Centric Learning?](https://mlsanity.com/paper/2504.07092/) (2025)
- [Scene-Centric Unsupervised Panoptic Segmentation](https://mlsanity.com/paper/2504.01955/) (2025)
- [Cycle Consistency Driven Object Discovery](https://mlsanity.com/paper/2306.02204/) (2023)
- [RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learning](https://mlsanity.com/paper/2308.14899/) (2023)
- [On Moving Object Segmentation from Monocular Video with Transformers](https://mlsanity.com/paper/2411.19141/) (2024)
- [Unsupervised Object Discovery: A Comprehensive Survey and Unified Taxonomy](https://mlsanity.com/paper/2411.00868/) (2024)
- [Slot-BERT: Self-supervised Object Discovery in Surgical Video](https://mlsanity.com/paper/2501.12477/) (2025)
- [Motion-Refined DINOSAUR for Unsupervised Multi-Object Discovery](https://mlsanity.com/paper/2509.02545/) (2025)
- [The Background Also Matters: Background-Aware Motion-Guided Objects Discovery](https://mlsanity.com/paper/2311.02633/) (2023)
- [Scene-Centric Unsupervised Video Panoptic Segmentation](https://mlsanity.com/paper/2606.04925/) (2026)
- [Variational Inference for Scalable 3D Object-centric Learning](https://mlsanity.com/paper/2309.14010/) (2023)
- [xMOD: Cross-Modal Distillation for 2D/3D Multi-Object Discovery from 2D motion](https://mlsanity.com/paper/2503.15022/) (2025)
- [Segmenting Collision Sound Sources in Egocentric Videos](https://mlsanity.com/paper/2511.13863/) (2025)

## Abstract (arXiv)

We propose a new approach to learn to segment multiple image objects without manual supervision. The method can extract objects form still images, but uses videos for supervision. While prior works have considered motion for segmentation, a key insight is that, while motion can be used to identify objects, not all objects are necessarily in motion: the absence of motion does not imply the absence of objects. Hence, our model learns to predict image regions that are likely to contain motion patterns characteristic of objects moving rigidly. It does not predict specific motion, which cannot be done unambiguously from a still image, but a distribution of possible motions, which includes the possibility that an object does not move at all. We demonstrate the advantage of this approach over its deterministic counterpart and show state-of-the-art unsupervised object segmentation performance on simulated and real-world benchmarks, surpassing methods that use motion even at test time. As our approach is applicable to variety of network architectures that segment the scenes, we also apply it to existing image reconstruction-based models showing drastic improvement. Project page and code: https://www.robots.ox.ac.uk/~vgg/research/ppmp .