# SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos

arXiv: https://arxiv.org/abs/2206.07764 · HTML: https://mlsanity.com/paper/2206.07764/

## Key facts

- Cited by 166 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 76 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 30 of 47)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [TrackFormer: Multi-Object Tracking with Transformers](https://mlsanity.com/paper/2101.02702/) (2021)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [TAO: A Large-Scale Benchmark for Tracking Any Object](https://mlsanity.com/paper/2005.10356/) (2020)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning](https://mlsanity.com/paper/1910.04744/) (2019)

## Built on by (most cited first, 30 of 166)

- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models](https://mlsanity.com/paper/2305.11281/) (2023)
- [Perceptual Grouping in Contrastive Vision-Language Models](https://mlsanity.com/paper/2210.09996/) (2022)
- [Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities](https://mlsanity.com/paper/2306.04829/) (2023)
- [Grounded Question-Answering in Long Egocentric Videos](https://mlsanity.com/paper/2312.06505/) (2023)
- [Improving Object-centric Learning with Query Optimization](https://mlsanity.com/paper/2210.08990/) (2022)
- [Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames](https://mlsanity.com/paper/2302.04973/) (2023)
- [An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learning](https://mlsanity.com/paper/2302.04419/) (2023)
- [Self-supervised Object-Centric Learning for Videos](https://mlsanity.com/paper/2310.06907/) (2023)
- [SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers](https://mlsanity.com/paper/2312.00648/) (2023)
- [Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiation](https://mlsanity.com/paper/2207.00787/) (2022)
- [Provably Learning Object-Centric Representations](https://mlsanity.com/paper/2305.14229/) (2023)
- [Provable Compositional Generalization for Object-Centric Learning](https://mlsanity.com/paper/2310.05327/) (2023)
- [Neural Systematic Binder](https://mlsanity.com/paper/2211.01177/) (2022)
- [Adaptive Slot Attention: Object Discovery with Dynamic Slot Number](https://mlsanity.com/paper/2406.09196/) (2024)
- [Helping Hands: An Object-Aware Ego-Centric Video Recognition Model](https://mlsanity.com/paper/2308.07918/) (2023)
- [Temporally Consistent Object-Centric Learning by Contrasting Slots](https://mlsanity.com/paper/2412.14295/) (2024)
- [OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environments](https://mlsanity.com/paper/2306.08649/) (2023)
- [Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models](https://mlsanity.com/paper/2406.09292/) (2024)
- [Scaling 4D Representations](https://mlsanity.com/paper/2412.15212/) (2024)
- [Rotating Features for Object Discovery](https://mlsanity.com/paper/2306.00600/) (2023)
- [OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulation](https://mlsanity.com/paper/2605.06481/) (2026)
- [Object Discovery from Motion-Guided Tokens](https://mlsanity.com/paper/2303.15555/) (2023)
- [How can objects help action recognition?](https://mlsanity.com/paper/2306.11726/) (2023)
- [Slot-VAE: Object-Centric Scene Generation with Slot Attention](https://mlsanity.com/paper/2306.06997/) (2023)
- [Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos](https://mlsanity.com/paper/2308.09951/) (2023)
- [Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective](https://mlsanity.com/paper/2511.11478/) (2025)
- [DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particles](https://mlsanity.com/paper/2306.05957/) (2023)
- [Zero-Shot Object-Centric Representation Learning](https://mlsanity.com/paper/2408.09162/) (2024)

## Abstract (arXiv)

The visual world can be parsimoniously characterized in terms of distinct entities with sparse interactions. Discovering this compositional structure in dynamic visual scenes has proven challenging for end-to-end computer vision approaches unless explicit instance-level supervision is provided. Slot-based models leveraging motion cues have recently shown great promise in learning to represent, segment, and track objects without direct supervision, but they still fail to scale to complex real-world multi-object videos. In an effort to bridge this gap, we take inspiration from human development and hypothesize that information about scene geometry in the form of depth signals can facilitate object-centric learning. We introduce SAVi++, an object-centric video model which is trained to predict depth signals from a slot-based video representation. By further leveraging best practices for model scaling, we are able to train SAVi++ to segment complex dynamic scenes recorded with moving cameras, containing both static and moving objects of diverse appearance on naturalistic backgrounds, without the need for segmentation supervision. Finally, we demonstrate that by using sparse depth signals obtained from LiDAR, SAVi++ is able to learn emergent object segmentation and tracking from videos in the real-world Waymo Open dataset.