# Segmenting Moving Objects via an Object-Centric Layered Representation

arXiv: https://arxiv.org/abs/2207.02206 · HTML: https://mlsanity.com/paper/2207.02206/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 91 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 30 of 53)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks](https://mlsanity.com/paper/2001.06810/) (2020)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- ["Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors](https://mlsanity.com/paper/1812.00467/) (2018)
- [MAST: A Memory-Augmented Self-supervised Tracker](https://mlsanity.com/paper/2002.07793/) (2020)
- [Semantic Amodal Segmentation](https://mlsanity.com/paper/1509.01329/) (2015)
- [Self-Supervised Scene De-occlusion](https://mlsanity.com/paper/2004.02788/) (2020)
- [Video Object Segmentation Without Temporal Information](https://mlsanity.com/paper/1709.06031/) (2017)
- [SegFlow: Joint Learning for Video Object Segmentation and Optical Flow](https://mlsanity.com/paper/1709.06750/) (2017)

## Built on by (most cited first, 30 of 63)

- [Easi3R: Estimating Disentangled Motion from DUSt3R Without Training](https://mlsanity.com/paper/2503.24391/) (2025)
- [Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities](https://mlsanity.com/paper/2306.04829/) (2023)
- [Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames](https://mlsanity.com/paper/2302.04973/) (2023)
- [Self-supervised Object-Centric Learning for Videos](https://mlsanity.com/paper/2310.06907/) (2023)
- [Advances in Deep Concealed Scene Understanding](https://mlsanity.com/paper/2304.11234/) (2023)
- [Amodal Ground Truth and Completion in the Wild](https://mlsanity.com/paper/2312.17247/) (2023)
- [VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentation](https://mlsanity.com/paper/2308.14710/) (2023)
- [A Survey of Camouflaged Object Detection and Beyond](https://mlsanity.com/paper/2408.14562/) (2024)
- [Segment Any Motion in Videos](https://mlsanity.com/paper/2503.22268/) (2025)
- [Moving Object Segmentation: All You Need Is SAM (and Flow)](https://mlsanity.com/paper/2404.12389/) (2024)
- [Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos](https://mlsanity.com/paper/2308.09951/) (2023)
- [RoMo: Robust Motion Segmentation Improves Structure from Motion](https://mlsanity.com/paper/2411.18650/) (2024)
- [Multi-Object Discovery by Low-Dimensional Object Motion](https://mlsanity.com/paper/2307.08027/) (2023)
- [Bootstrapping Objectness from Videos by Relaxed Common Fate and Visual Grouping](https://mlsanity.com/paper/2304.08025/) (2023)
- [Learning segmentation from point trajectories](https://mlsanity.com/paper/2501.12392/) (2025)
- [Motion-inductive Self-supervised Object Discovery in Videos](https://mlsanity.com/paper/2210.00221/) (2022)
- [The Making and Breaking of Camouflage](https://mlsanity.com/paper/2309.03899/) (2023)
- [Appearance-Based Refinement for Object-Centric Motion Segmentation](https://mlsanity.com/paper/2312.11463/) (2023)
- [Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection](https://mlsanity.com/paper/2403.01968/) (2024)
- [Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation](https://mlsanity.com/paper/2311.17893/) (2023)
- [4D-VGGT: A General Foundation Model with SpatioTemporal Awareness for Dynamic Scene Geometry Estimation](https://mlsanity.com/paper/2511.18416/) (2025)
- [Event-Free Moving Object Segmentation from Moving Ego Vehicle](https://mlsanity.com/paper/2305.00126/) (2023)
- [LOCATE: Self-supervised Object Discovery via Flow-guided Graph-cut and Bootstrapped Self-training](https://mlsanity.com/paper/2308.11239/) (2023)
- [Amodal Optical Flow](https://mlsanity.com/paper/2311.07761/) (2023)
- [Rethinking Image-to-Video Adaptation: An Object-centric Perspective](https://mlsanity.com/paper/2407.06871/) (2024)
- [Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes](https://mlsanity.com/paper/2403.04562/) (2024)
- [Reasoning-Enhanced Object-Centric Learning for Videos](https://mlsanity.com/paper/2403.15245/) (2024)
- [CamSAM2: Segment Anything Accurately in Camouflaged Videos](https://mlsanity.com/paper/2503.19730/) (2025)
- [On Moving Object Segmentation from Monocular Video with Transformers](https://mlsanity.com/paper/2411.19141/) (2024)
- [Made to Order: Discovering monotonic temporal changes via self-supervised video ordering](https://mlsanity.com/paper/2404.16828/) (2024)

## Abstract (arXiv)

The objective of this paper is a model that is able to discover, track and segment multiple moving objects in a video. We make four contributions: First, we introduce an object-centric segmentation model with a depth-ordered layer representation. This is implemented using a variant of the transformer architecture that ingests optical flow, where each query vector specifies an object and its layer for the entire video. The model can effectively discover multiple moving objects and handle mutual occlusions; Second, we introduce a scalable pipeline for generating multi-object synthetic training data via layer compositions, that is used to train the proposed model, significantly reducing the requirements for labour-intensive annotations, and supporting Sim2Real generalisation; Third, we conduct thorough ablation studies, showing that the model is able to learn object permanence and temporal shape consistency, and is able to predict amodal segmentation masks; Fourth, we evaluate our model, trained only on synthetic data, on standard video segmentation benchmarks, DAVIS, MoCA, SegTrack, FBMS-59, and achieve state-of-the-art performance among existing methods that do not rely on any manual annotations. With test-time adaptation, we observe further performance boosts.