# Discovering Objects that Can Move

arXiv: https://arxiv.org/abs/2203.10159 · HTML: https://mlsanity.com/paper/2203.10159/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 70 references that are in the map.
- First posted on arXiv 18 Mar 2022.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [End-to-end people detection in crowded scenes](https://mlsanity.com/paper/1506.04878/) (2015)
- [Learning Features by Watching Objects Move](https://mlsanity.com/paper/1612.06370/) (2016)
- [CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning](https://mlsanity.com/paper/1910.04744/) (2019)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)

## Built on by (most cited first, 30 of 44)

- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [Perceptual Grouping in Contrastive Vision-Language Models](https://mlsanity.com/paper/2210.09996/) (2022)
- [Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities](https://mlsanity.com/paper/2306.04829/) (2023)
- [Self-supervised Object-Centric Learning for Videos](https://mlsanity.com/paper/2310.06907/) (2023)
- [Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion](https://mlsanity.com/paper/2205.07844/) (2022)
- [AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2403.17373/) (2024)
- [Object Discovery from Motion-Guided Tokens](https://mlsanity.com/paper/2303.15555/) (2023)
- [Understanding Video Transformers via Universal Concept Discovery](https://mlsanity.com/paper/2401.10831/) (2024)
- [Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns](https://mlsanity.com/paper/2210.12148/) (2022)
- [Neural Groundplans: Persistent Neural Scene Representations from a Single Image](https://mlsanity.com/paper/2207.11232/) (2022)
- [Boosting Object Representation Learning via Motion and Object Continuity](https://mlsanity.com/paper/2211.09771/) (2022)
- [Efficient Image Pre-Training with Siamese Cropped Masked Autoencoders](https://mlsanity.com/paper/2403.17823/) (2024)
- [Object-centric Learning with Cyclic Walks between Parts and Whole](https://mlsanity.com/paper/2302.08023/) (2023)
- [Multi-Object Discovery by Low-Dimensional Object Motion](https://mlsanity.com/paper/2307.08027/) (2023)
- [Object-Centric World Model for Language-Guided Manipulation](https://mlsanity.com/paper/2503.06170/) (2025)
- [Object-Centric Multiple Object Tracking](https://mlsanity.com/paper/2309.00233/) (2023)
- [Motion-inductive Self-supervised Object Discovery in Videos](https://mlsanity.com/paper/2210.00221/) (2022)
- [Unsupervised Open-Vocabulary Object Localization in Videos](https://mlsanity.com/paper/2309.09858/) (2023)
- [Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes](https://mlsanity.com/paper/2311.17948/) (2023)
- [Appearance-Based Refinement for Object-Centric Motion Segmentation](https://mlsanity.com/paper/2312.11463/) (2023)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [CarFormer: Self-Driving with Learned Object-Centric Representations](https://mlsanity.com/paper/2407.15843/) (2024)
- [Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes](https://mlsanity.com/paper/2403.04562/) (2024)
- [Highly Efficient and Unsupervised Framework for Moving Object Detection in Satellite Videos](https://mlsanity.com/paper/2411.15895/) (2024)
- [RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in Object-centric Learning](https://mlsanity.com/paper/2308.14899/) (2023)
- [On Moving Object Segmentation from Monocular Video with Transformers](https://mlsanity.com/paper/2411.19141/) (2024)
- [Instance-Level Moving Object Segmentation from a Single Image with Events](https://mlsanity.com/paper/2502.12975/) (2025)
- [WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction](https://mlsanity.com/paper/2211.14308/) (2022)

## Abstract (arXiv)

This paper studies the problem of object discovery -- separating objects from the background without manual labels. Existing approaches utilize appearance cues, such as color, texture, and location, to group pixels into object-like regions. However, by relying on appearance alone, these methods fail to separate objects from the background in cluttered scenes. This is a fundamental limitation since the definition of an object is inherently ambiguous and context-dependent. To resolve this ambiguity, we choose to focus on dynamic objects -- entities that can move independently in the world. We then scale the recent auto-encoder based frameworks for unsupervised object discovery from toy synthetic images to complex real-world scenes. To this end, we simplify their architecture, and augment the resulting model with a weak learning signal from general motion segmentation algorithms. Our experiments demonstrate that, despite only capturing a small subset of the objects that move, this signal is enough to generalize to segment both moving and static instances of dynamic objects. We show that our model scales to a newly collected, photo-realistic synthetic dataset with street driving scenarios. Additionally, we leverage ground truth segmentation and flow annotations in this dataset for thorough ablation and evaluation. Finally, our experiments on the real-world KITTI benchmark demonstrate that the proposed approach outperforms both heuristic- and learning-based methods by capitalizing on motion cues.