mlsanityData to 14 Oct 2026

SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos

arXiv 2206.07764 ↗First posted 15 Jun 2022cs.CV cs.LGcited by 166

Not on the map. Why?

Abstract (arXiv)

The visual world can be parsimoniously characterized in terms of distinct entities with sparse interactions. Discovering this compositional structure in dynamic visual scenes has proven challenging for end-to-end computer vision approaches unless explicit instance-level supervision is provided. Slot-based models leveraging motion cues have recently shown great promise in learning to represent, segment, and track objects without direct supervision, but they still fail to scale to complex real-world multi-object videos. In an effort to bridge this gap, we take inspiration from human development and hypothesize that information about scene geometry in the form of depth signals can facilitate object-centric learning. We introduce SAVi++, an object-centric video model which is trained to predict depth signals from a slot-based video representation. By further leveraging best practices for model scaling, we are able to train SAVi++ to segment complex dynamic scenes recorded with moving cameras, containing both static and moving objects of diverse appearance on naturalistic backgrounds, without the need for segmentation supervision. Finally, we demonstrate that by using sparse depth signals obtained from LiDAR, SAVi++ is able to learn emergent object segmentation and tracking from videos in the real-world Waymo Open dataset.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). Builds on 47 of its 76 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

47 of 76 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2014Going Deeper with Convolutionscited by 4,241
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2021Vision Transformers for Dense Predictioncited by 1,803
  11. 2018Group Normalizationcited by 1,524
  12. 2017Deep Image Priorcited by 1,098
35 more
  1. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  2. 2020On Layer Normalization in the Transformer Architecturecited by 922
  3. 2020Object-Centric Learning with Slot Attentioncited by 876
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  5. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  6. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
  7. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  8. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 518
  9. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  10. 2022Kubric: A scalable dataset generatorcited by 381
  11. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  12. 2021Conditional Object-Centric Learning from Videocited by 248
  13. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  14. 2020On the Binding Problem in Artificial Neural Networkscited by 233
  15. 2019Contrastive Learning of Structured World Modelscited by 209
  16. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 147
  17. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  18. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  19. 2018Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objectscited by 128
  20. 2017Neural Expectation Maximizationcited by 123
  21. 2019Entity Abstraction in Visual Model-Based Reinforcement Learningcited by 110
  22. 2019SCALOR: Generative World Models with Scalable Object Representationscited by 108
  23. 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
  24. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 97
  25. 2021ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentationcited by 93
  26. 2020A Survey on Deep Learning Techniques for Stereo-based Depth Estimationcited by 90
  27. 2022Object discovery and representation networkscited by 80
  28. 2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
  29. 2020Learning Physical Graph Representations from Visual Scenescited by 50
  30. 2022Discovering Objects that Can Movecited by 44
  31. 2021Slot-VPS: Object-centric Representation Learning for Video Panoptic Segmentationcited by 24
  32. 2020Unsupervised Object Detection with LiDAR Cluescited by 22
  33. 2020Learning Object-Centric Video Models by Contrasting Setscited by 19
  34. 2022Drive&Segment: Unsupervised Semantic Segmentation of Urban Scenes via Cross-modal Distillationcited by 19
  35. 2021Track, Check, Repeat: An EM Approach to Unsupervised Trackingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 44; 2024: 44; 2025: 35; 2026: 34.

166 papers.

Sort:

By area:

  1. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 220
  2. 2022Disentangled Representation Learningcited by 112
  3. 2023SlotDiffusion: Object-Centric Generative Modeling with Diffusion Modelscited by 85
  4. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  5. 2023Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similaritiescited by 78
  6. 2023Grounded Question-Answering in Long Egocentric Videoscited by 64
  7. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  8. 2023Invariant Slot Attention: Object Discovery with Slot-Centric Reference Framescited by 51
  9. 2023An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learningcited by 49
  10. 2023Self-supervised Object-Centric Learning for Videoscited by 49
  11. 2023SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformerscited by 49
  12. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
38 more
  1. 2023Provably Learning Object-Centric Representationscited by 45
  2. 2023Provable Compositional Generalization for Object-Centric Learningcited by 41
  3. 2022Neural Systematic Bindercited by 37
  4. 2024Adaptive Slot Attention: Object Discovery with Dynamic Slot Numbercited by 34
  5. 2023Helping Hands: An Object-Aware Ego-Centric Video Recognition Modelcited by 31
  6. 2024Temporally Consistent Object-Centric Learning by Contrasting Slotscited by 31
  7. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  8. 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
  9. 2024Scaling 4D Representationscited by 30
  10. 2023Rotating Features for Object Discoverycited by 29
  11. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 28
  12. 2023Object Discovery from Motion-Guided Tokenscited by 25
  13. 2023How can objects help action recognition?cited by 21
  14. 2023Slot-VAE: Object-Centric Scene Generation with Slot Attentioncited by 20
  15. 2023Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videoscited by 20
  16. 2025Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspectivecited by 20
  17. 2023DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particlescited by 19
  18. 2024Zero-Shot Object-Centric Representation Learningcited by 19
  19. 2022Boosting Object Representation Learning via Motion and Object Continuitycited by 17
  20. 2023Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learningcited by 17
  21. 2023Grounded Object Centric Learningcited by 17
  22. 2024Slot State Space Modelscited by 17
  23. 2023Object-centric Learning with Cyclic Walks between Parts and Wholecited by 16
  24. 2023Contrastive Training of Complex-Valued Autoencoders for Object Discoverycited by 16
  25. 2023Multi-Object Discovery by Low-Dimensional Object Motioncited by 16
  26. 2025Object-Centric World Model for Language-Guided Manipulationcited by 16
  27. 2023Object-Centric Multiple Object Trackingcited by 15
  28. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
  29. 2024Neural Concept Bindercited by 14
  30. 2024Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Modelscited by 14
  31. 2024GLASS: Guided Latent Slot Diffusion for Object-Centric Learningcited by 14
  32. 2025PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planningcited by 14
  33. 2025Slot Attention with Re-Initialization and Self-Distillationcited by 14
  34. 2023Semantic Attention Flow Fields for Monocular Dynamic Scene Decompositioncited by 13
  35. 2023Unsupervised Open-Vocabulary Object Localization in Videoscited by 13
  36. 2025Slot-Guided Adaptation of Pre-trained Diffusion Models for Object-Centric Learning and Compositional Generationcited by 13
  37. 2025Dyn-O: Building Structured World Models with Object-Centric Representationscited by 13
  38. 2025Dissecting Generalized Category Discovery: Multiplex Consensus under Self-Deconstructioncited by 13

50 most cited of 166 All 166 (JSON)