# Unsupervised Object-Based Transition Models for 3D Partially Observable Environments

arXiv: https://arxiv.org/abs/2103.04693 · HTML: https://mlsanity.com/paper/2103.04693/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 8 Mar 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [Learning to Decompose and Disentangle Representations for Video Prediction](https://mlsanity.com/paper/1806.04166/) (2018)
- [Entity Abstraction in Visual Model-Based Reinforcement Learning](https://mlsanity.com/paper/1910.12827/) (2019)
- [Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions](https://mlsanity.com/paper/1802.10353/) (2018)
- [Tagger: Deep Unsupervised Perceptual Grouping](https://mlsanity.com/paper/1606.06724/) (2016)
- [Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text](https://mlsanity.com/paper/2005.09382/) (2020)
- [COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration](https://mlsanity.com/paper/1905.09275/) (2019)
- [Environmental drivers of systematicity and generalization in a situated agent](https://mlsanity.com/paper/1910.00571/) (2019)
- [Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systems](https://mlsanity.com/paper/2006.16225/) (2020)
- [Towards Deep Symbolic Reinforcement Learning](https://mlsanity.com/paper/1609.05518/) (2016)
- [Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers](https://mlsanity.com/paper/1809.03137/) (2018)
- [Learning Object-Centric Video Models by Contrasting Sets](https://mlsanity.com/paper/2011.10287/) (2020)
- [AlignNet: Unsupervised Entity Alignment](https://mlsanity.com/paper/2007.08973/) (2020)

## Built on by (most cited first, 20 of 20)

- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [TransDreamer: Reinforcement Learning with Transformer World Models](https://mlsanity.com/paper/2202.09481/) (2022)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models](https://mlsanity.com/paper/2305.11281/) (2023)
- [DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation](https://mlsanity.com/paper/2308.07498/) (2023)
- [SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition](https://mlsanity.com/paper/2106.03849/) (2021)
- [Neural Systematic Binder](https://mlsanity.com/paper/2211.01177/) (2022)
- [Generative Video Transformer: Can Objects be the Words?](https://mlsanity.com/paper/2107.09240/) (2021)
- [Toward Compositional Generalization in Object-Oriented World Modeling](https://mlsanity.com/paper/2204.13661/) (2022)
- [Abstraction for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.05839/) (2022)
- [Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactions](https://mlsanity.com/paper/2302.11850/) (2023)
- [PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning](https://mlsanity.com/paper/2502.07600/) (2025)
- [Learning to Generalize with Object-centric Agents in the Open World Survival Game Crafter](https://mlsanity.com/paper/2208.03374/) (2022)
- [VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE](https://mlsanity.com/paper/2401.11110/) (2024)
- [Towards Improving the Generation Quality of Autoregressive Slot VAEs](https://mlsanity.com/paper/2206.01370/) (2022)
- [Learning Disentangled Representation in Object-Centric Models for Visual Dynamics Prediction via Transformers](https://mlsanity.com/paper/2407.03216/) (2024)
- [TextOCVP: Object-Centric Video Prediction with Language Guidance](https://mlsanity.com/paper/2502.11655/) (2025)
- [Efficient entity-based reinforcement learning](https://mlsanity.com/paper/2206.02855/) (2022)
- [Grid or Nodes? Probing Object Memory in World Models Under Partial Observability](https://mlsanity.com/paper/2610.18637/) (2026)

## Abstract (arXiv)

We present a slot-wise, object-based transition model that decomposes a scene into objects, aligns them (with respect to a slot-wise object memory) to maintain a consistent order across time, and predicts how those objects evolve over successive frames. The model is trained end-to-end without supervision using losses at the level of the object-structured representation rather than pixels. Thanks to its alignment module, the model deals properly with two issues that are not handled satisfactorily by other transition models, namely object persistence and object identity. We show that the combination of an object-level loss and correct object alignment over time enables the model to outperform a state-of-the-art baseline, and allows it to deal well with object occlusion and re-appearance in partially observable environments.