mlsanityData to 14 Oct 2026

Conditional Object-Centric Learning from Video

arXiv 2111.12594 ↗First posted 24 Nov 2021cs.CV cs.LG stat.MLcited by 248

On the map: World models → Latent predictive world models

Abstract (arXiv)

Object-centric representations are a promising path toward more systematic generalization by providing flexible abstractions upon which compositional world models can be built. Recent work on simple 2D and 3D datasets has shown that models with object-centric inductive biases can learn to segment and represent meaningful objects from the statistical structure of the data alone without the need for any supervision. However, such fully-unsupervised methods still fail to scale to diverse realistic data, despite the use of increasingly complex inductive biases such as priors for the size of objects or the 3D geometry of the scene. In this paper, we instead take a weakly-supervised approach and focus on how 1) using the temporal dynamics of video data in the form of optical flow and 2) conditioning the model on simple object location cues can be used to enable segmenting and tracking objects in significantly more realistic synthetic data. We introduce a sequential extension to Slot Attention which we train to predict optical flow for realistic looking synthetic scenes and show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the first frame, is sufficient to significantly improve instance segmentation. These benefits generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences. We also find that such initial-state-conditioning can be used during inference as a flexible interface to query the model for specific objects or parts of objects, which could pave the way for a range of weakly-supervised approaches and allow more effective interaction with trained models.

Key facts. Cited by 248 papers in the map (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 69 of its 89 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

69 of 89 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Contrastive Learning of Structured World Models27 Nov
  2. 2019SCALOR: Generative World Models with Scalable Object Representations6 Oct

Other references · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2016Layer Normalizationcited by 5,669
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  9. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  10. 2018Group Normalizationcited by 1,524
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  12. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
38 more
  1. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  2. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  3. 2020Object-Centric Learning with Slot Attentioncited by 876
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  5. 2017Dynamic Routing Between Capsulescited by 689
  6. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  7. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 518
  8. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  9. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  10. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  11. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  12. 2018Video Action Transformer Networkcited by 316
  13. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  14. 2016One-Shot Video Object Segmentationcited by 235
  15. 2015Structural-RNN: Deep Learning on Spatio-Temporal Graphscited by 234
  16. 2020On the Binding Problem in Artificial Neural Networkscited by 233
  17. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  18. 2019Recurrent Independent Mechanismscited by 219
  19. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  20. 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
  21. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  22. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  23. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  24. 2017Neural Expectation Maximizationcited by 123
  25. 2019Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEscited by 117
  26. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  27. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  28. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 97
  29. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  30. 2021Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentationcited by 88
  31. 2018PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentationcited by 81
  32. 2019Joint-task Self-supervised Learning for Temporal Correspondencecited by 80
  33. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  34. 2021SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warpingcited by 74
  35. 2021Unsupervised Learning of Compositional Energy Conceptscited by 71
  36. 2016Tagger: Deep Unsupervised Perceptual Groupingcited by 69
  37. 2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
  38. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 46; 2023: 59; 2024: 50; 2025: 38; 2026: 55.

248 papers.

Sort:

By area:

  1. 2022Kubric: A scalable dataset generatorcited by 381
  2. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 220
  3. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 166
  4. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  5. 2022Weakly supervised causal representation learningcited by 140
  6. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  7. 2022Object Scene Representation Transformercited by 90
  8. 2023Object-Centric Slot Diffusioncited by 90
  9. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  10. 2022Object discovery and representation networkscited by 80
  11. 2023Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similaritiescited by 78
  12. 2022Self-Supervised Visual Representation Learning with Semantic Groupingcited by 70
38 more
  1. 2023Compositional Generalization from First Principlescited by 64
  2. 2022Segmenting Moving Objects via an Object-Centric Layered Representationcited by 63
  3. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  4. 2022Unsupervised Semantic Segmentation with Self-supervised Object-centric Representationscited by 54
  5. 2022Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systemscited by 51
  6. 2023Invariant Slot Attention: Object Discovery with Slot-Centric Reference Framescited by 51
  7. 2023An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learningcited by 49
  8. 2023Self-supervised Object-Centric Learning for Videoscited by 49
  9. 2023SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformerscited by 49
  10. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  11. 2023Provably Learning Object-Centric Representationscited by 45
  12. 2022Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motioncited by 42
  13. 2022Complex-Valued Autoencoders for Object Discoverycited by 40
  14. 2022Neural Systematic Bindercited by 37
  15. 2023Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representationscited by 35
  16. 2024Adaptive Slot Attention: Object Discovery with Dynamic Slot Numbercited by 34
  17. 2022VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programmingcited by 31
  18. 2024Temporally Consistent Object-Centric Learning by Contrasting Slotscited by 31
  19. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  20. 2024Scaling 4D Representationscited by 30
  21. 2023Rotating Features for Object Discoverycited by 29
  22. 2026Causal-JEPA: Learning World Models through Object-Level Latent Maskingcited by 29
  23. 2022Unsupervised Discovery and Composition of Object Light Fieldscited by 28
  24. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 28
  25. 2022Compositional Scene Representation Learning via Reconstruction: A Surveycited by 27
  26. 2022OGC: Unsupervised 3D Object Segmentation from Rigid Dynamics of Point Cloudscited by 25
  27. 2023Object Discovery from Motion-Guided Tokenscited by 25
  28. 2024From an Image to a Scene: Learning to Imagine the World from a Million 360 Videoscited by 23
  29. 2022Toward Compositional Generalization in Object-Oriented World Modelingcited by 22
  30. 2022Visuomotor Control in Multi-Object Scenes Using Object-Aware Representationscited by 21
  31. 2022Learning What and Where: Disentangling Location and Identity Tracking Without Supervisioncited by 21
  32. 2023Mask-Free Video Instance Segmentationcited by 20
  33. 2023Slot-VAE: Object-Centric Scene Generation with Slot Attentioncited by 20
  34. 2023Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videoscited by 20
  35. 2022Compositional Multi-Object Reinforcement Learning with Linear Relation Networkscited by 19
  36. 2022Unsupervised Multi-object Segmentation by Predicting Probable Motion Patternscited by 19
  37. 2023Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactionscited by 19
  38. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 19

50 most cited of 248 All 248 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Object-Centric Representations Based on Slots in Real World Scenarios
  2. 2024Using Diffusion Priors for Video Amodal Segmentation
  3. 2025Conditional Latent Diffusion Models for Zero-Shot Instance Segmentation
  4. 2025Oh-A-DINO: Understanding and Enhancing Attribute-Level Information in Self-Supervised Object-Centric Representations
  5. 2023DORSal: Diffusion for Object-centric Representations of Scenes et al
  6. 2026HiddenObjects: Scalable Diffusion-Distilled Spatial Priors for Object Placement
  7. 2024Learning to Compose: Improving Object Centric Learning by Injecting Compositionality
  8. 2022Is an Object-Centric Video Representation Beneficial for Transfer?