# Learning Object Permanence from Videos via Latent Imaginations

arXiv: https://arxiv.org/abs/2310.10372 · HTML: https://mlsanity.com/paper/2310.10372/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 53 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [DayDreamer: World Models for Physical Robot Learning](https://mlsanity.com/paper/2206.14176/) (2022)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning](https://mlsanity.com/paper/1910.04744/) (2019)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)
- [Improving Generative Imagination in Object-Centric World Models](https://mlsanity.com/paper/2010.02054/) (2020)
- [Attention over learned object embeddings enables complex visual reasoning](https://mlsanity.com/paper/2012.08508/) (2020)
- [Structured Object-Aware Physics Prediction for Video Modeling and Planning](https://mlsanity.com/paper/1910.02425/) (2019)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [Learning What and Where: Disentangling Location and Identity Tracking Without Supervision](https://mlsanity.com/paper/2205.13349/) (2022)
- [Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets](https://mlsanity.com/paper/1901.09049/) (2019)
- [Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains](https://mlsanity.com/paper/2110.15949/) (2021)

## Built on by (most cited first, 4 of 4)

- [Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation](https://mlsanity.com/paper/2502.02763/) (2025)
- [Event-Conditioned Diagnostics of Kinematic, Contact, and Object-Permanence Structure in Passive Object-State World Models](https://mlsanity.com/paper/2606.28455/) (2026)
- [iSEE: Object Permanence Through Self-Supervision](https://mlsanity.com/paper/2610.01201/) (2026)
- [EVEWorld: Physical Evolution Supervision for Embodied World Models](https://mlsanity.com/paper/2610.03374/) (2026)

## Abstract (arXiv)

While human infants exhibit knowledge about object permanence from two months of age onwards, deep-learning approaches still largely fail to recognize objects' continued existence. We introduce a slot-based autoregressive deep learning system, the looped location and identity tracking model Loci-Looped, which learns to adaptively fuse latent imaginations with pixel-space observations into consistent latent object-specific what and where encodings over time. The novel loop empowers Loci-Looped to learn the physical concepts of object permanence, directional inertia, and object solidity through observation alone. As a result, Loci-Looped tracks objects through occlusions, anticipates their reappearance, and shows signs of surprise and internal revisions when observing implausible object behavior. Notably, Loci-Looped outperforms state-of-the-art baseline models in handling object occlusions and temporary sensory interruptions while exhibiting more compositional, interpretable internal activity patterns. Our work thus introduces the first self-supervised interpretable learning model that learns about object permanence directly from video data without supervision.