# Benchmarking Unsupervised Object Representations for Video Sequences

arXiv: https://arxiv.org/abs/2006.07034 · HTML: https://mlsanity.com/paper/2006.07034/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 27 of 27)

- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities](https://mlsanity.com/paper/2306.04829/) (2023)
- [SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers](https://mlsanity.com/paper/2312.00648/) (2023)
- [Provably Learning Object-Centric Representations](https://mlsanity.com/paper/2305.14229/) (2023)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [Learning What and Where: Disentangling Location and Identity Tracking Without Supervision](https://mlsanity.com/paper/2205.13349/) (2022)
- [Unsupervised Multi-object Segmentation by Predicting Probable Motion Patterns](https://mlsanity.com/paper/2210.12148/) (2022)
- [DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particles](https://mlsanity.com/paper/2306.05957/) (2023)
- [Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images](https://mlsanity.com/paper/2210.02324/) (2022)
- [Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling](https://mlsanity.com/paper/2603.04553/) (2026)
- [Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models](https://mlsanity.com/paper/2407.15589/) (2024)
- [Unsupervised Learning of Compositional Scene Representations from Multiple Unspecified Viewpoints](https://mlsanity.com/paper/2112.03568/) (2021)
- [Appearance-Based Refinement for Object-Centric Motion Segmentation](https://mlsanity.com/paper/2312.11463/) (2023)
- [Intrinsic Physical Concepts Discovery with Object-Centric Predictive Models](https://mlsanity.com/paper/2303.01869/) (2023)
- [Spotlight Attention: Robust Object-Centric Learning With a Spatial Locality Prior](https://mlsanity.com/paper/2305.19550/) (2023)
- [Unsupervised Image Decomposition with Phase-Correlation Networks](https://mlsanity.com/paper/2110.03473/) (2021)
- [Object-Category Aware Reinforcement Learning](https://mlsanity.com/paper/2210.07802/) (2022)
- [SlotPi: Physics-informed Object-centric Reasoning Models](https://mlsanity.com/paper/2506.10778/) (2025)
- [Benchmarking and Analysis of Unsupervised Object Segmentation from Real-world Single Images](https://mlsanity.com/paper/2312.04947/) (2023)
- [Unsupervised Object-Centric Learning from Multiple Unspecified Viewpoints](https://mlsanity.com/paper/2401.01922/) (2024)
- [unMORE: Unsupervised Multi-Object Segmentation via Center-Boundary Reasoning](https://mlsanity.com/paper/2506.01778/) (2025)
- [Pixel-level Correspondence for Self-Supervised Learning from Video](https://mlsanity.com/paper/2207.03866/) (2022)
- [Enhancing Interpretable Object Abstraction via Clustering-based Slot Initialization](https://mlsanity.com/paper/2308.11369/) (2023)
- [VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE](https://mlsanity.com/paper/2401.11110/) (2024)
- [Ablation Study to Clarify the Mechanism of Object Segmentation in Multi-Object Representation Learning](https://mlsanity.com/paper/2310.03273/) (2023)
- [VideoPCDNet: Video Parsing and Prediction with Phase Correlation Networks](https://mlsanity.com/paper/2506.19621/) (2025)
- [ContextFusion and Bootstrap: An Effective Approach to Improve Slot Attention-Based Object-Centric Learning](https://mlsanity.com/paper/2509.02032/) (2025)

## Abstract (arXiv)

Perceiving the world in terms of objects and tracking them through time is a crucial prerequisite for reasoning and scene understanding. Recently, several methods have been proposed for unsupervised learning of object-centric representations. However, since these models were evaluated on different downstream tasks, it remains unclear how they compare in terms of basic perceptual abilities such as detection, figure-ground segmentation and tracking of objects. To close this gap, we design a benchmark with four data sets of varying complexity and seven additional test sets featuring challenging tracking scenarios relevant for natural videos. Using this benchmark, we compare the perceptual abilities of four object-centric approaches: ViMON, a video-extension of MONet, based on recurrent spatial attention, OP3, which exploits clustering via spatial mixture models, as well as TBA and SCALOR, which use explicit factorization via spatial transformers. Our results suggest that the architectures with unconstrained latent representations learn more powerful representations in terms of object detection, segmentation and tracking than the spatial transformer based architectures. We also observe that none of the methods are able to gracefully handle the most challenging tracking scenarios despite their synthetic nature, suggesting that our benchmark may provide fruitful guidance towards learning more robust object-centric video representations.