# Self-supervised Object-Centric Learning for Videos

arXiv: https://arxiv.org/abs/2310.06907 · HTML: https://mlsanity.com/paper/2310.06907/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 69 of its 89 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 30 of 69)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [Video Object Segmentation using Space-Time Memory Networks](https://mlsanity.com/paper/1904.00607/) (2019)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark](https://mlsanity.com/paper/1809.03327/) (2018)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Cut and Learn for Unsupervised Object Detection and Instance Segmentation](https://mlsanity.com/paper/2301.11320/) (2023)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)

## Built on by (most cited first, 30 of 49)

- [SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers](https://mlsanity.com/paper/2312.00648/) (2023)
- [Temporally Consistent Object-Centric Learning by Contrasting Slots](https://mlsanity.com/paper/2412.14295/) (2024)
- [Causal-JEPA: Learning World Models through Object-Level Latent Masking](https://mlsanity.com/paper/2602.11389/) (2026)
- [Zero-Shot Object-Centric Representation Learning](https://mlsanity.com/paper/2408.09162/) (2024)
- [Object-Centric World Model for Language-Guided Manipulation](https://mlsanity.com/paper/2503.06170/) (2025)
- [Learning segmentation from point trajectories](https://mlsanity.com/paper/2501.12392/) (2025)
- [Can Visual Foundation Models Achieve Long-term Point Tracking?](https://mlsanity.com/paper/2408.13575/) (2024)
- [PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning](https://mlsanity.com/paper/2502.07600/) (2025)
- [Slot Attention with Re-Initialization and Self-Distillation](https://mlsanity.com/paper/2507.23755/) (2025)
- [Dyn-O: Building Structured World Models with Object-Centric Representations](https://mlsanity.com/paper/2507.03298/) (2025)
- [Dissecting Generalized Category Discovery: Multiplex Consensus under Self-Deconstruction](https://mlsanity.com/paper/2508.10731/) (2025)
- [A Likelihood Ratio-Based Approach to Segmenting Unknown Objects](https://mlsanity.com/paper/2409.06424/) (2024)
- [Appearance-Based Refinement for Object-Centric Motion Segmentation](https://mlsanity.com/paper/2312.11463/) (2023)
- [MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning](https://mlsanity.com/paper/2505.20772/) (2025)
- [Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation](https://mlsanity.com/paper/2311.17893/) (2023)
- [Rethinking Image-to-Video Adaptation: An Object-centric Perspective](https://mlsanity.com/paper/2407.06871/) (2024)
- [CarFormer: Self-Driving with Learned Object-Centric Representations](https://mlsanity.com/paper/2407.15843/) (2024)
- [Robust Bird's Eye View Segmentation by Adapting DINOv2](https://mlsanity.com/paper/2409.10228/) (2024)
- [OCRT: Boosting Foundation Models in the Open World with Object-Concept-Relation Triad](https://mlsanity.com/paper/2503.18695/) (2025)
- [Smoothing Slot Attention Iterations and Recurrences](https://mlsanity.com/paper/2508.05417/) (2025)
- [From Vicious to Virtuous Cycles: Synergistic Representation Learning for Unsupervised Video Object-Centric Learning](https://mlsanity.com/paper/2602.03390/) (2026)
- [SPARO: Selective Attention for Robust and Compositional Transformer Encodings for Vision](https://mlsanity.com/paper/2404.15721/) (2024)
- [Slot-BERT: Self-supervised Object Discovery in Surgical Video](https://mlsanity.com/paper/2501.12477/) (2025)
- [unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning](https://mlsanity.com/paper/2506.01778/) (2025)
- [Motion-Refined DINOSAUR for Unsupervised Multi-Object Discovery](https://mlsanity.com/paper/2509.02545/) (2025)
- [Reconstruction-Guided Slot Curriculum: Addressing Object Over-Fragmentation in Video Object-Centric Learning](https://mlsanity.com/paper/2603.22758/) (2026)
- [MOD-UV: Learning Mobile Object Detectors from Unlabeled Videos](https://mlsanity.com/paper/2405.14841/) (2024)
- [FlowCut: Unsupervised Video Instance Segmentation via Temporal Mask Matching](https://mlsanity.com/paper/2505.13174/) (2025)
- [FRAME: Pre-Training Video Feature Representations via Anticipation and Memory](https://mlsanity.com/paper/2506.05543/) (2025)
- [Future Slot Prediction for Unsupervised Object Discovery in Surgical Video](https://mlsanity.com/paper/2507.01882/) (2025)

## Abstract (arXiv)

Unsupervised multi-object segmentation has shown impressive results on images by utilizing powerful semantics learned from self-supervised pretraining. An additional modality such as depth or motion is often used to facilitate the segmentation in video sequences. However, the performance improvements observed in synthetic sequences, which rely on the robustness of an additional cue, do not translate to more challenging real-world scenarios. In this paper, we propose the first fully unsupervised method for segmenting multiple objects in real-world sequences. Our object-centric learning framework spatially binds objects to slots on each frame and then relates these slots across frames. From these temporally-aware slots, the training objective is to reconstruct the middle frame in a high-level semantic feature space. We propose a masking strategy by dropping a significant portion of tokens in the feature space for efficiency and regularization. Additionally, we address over-clustering by merging slots based on similarity. Our method can successfully segment multiple instances of complex and high-variety classes in YouTube videos.