# Time-Conditioned Generative Modeling of Object-Centric Representations for Video Decomposition and Prediction

arXiv: https://arxiv.org/abs/2301.08951 · HTML: https://mlsanity.com/paper/2301.08951/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 21 Jan 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement](https://mlsanity.com/paper/2104.09958/) (2021)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)
- [Improving Generative Imagination in Object-Centric World Models](https://mlsanity.com/paper/2010.02054/) (2020)
- [SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition](https://mlsanity.com/paper/2106.03849/) (2021)
- [ROOTS: Object-Centric Representation and Rendering of 3D Scenes](https://mlsanity.com/paper/2006.06130/) (2020)
- [Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows](https://mlsanity.com/paper/2002.10516/) (2020)
- [Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views](https://mlsanity.com/paper/2111.07117/) (2021)
- [Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations](https://mlsanity.com/paper/2106.03630/) (2021)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [SHOP-VRB: A Visual Reasoning Benchmark for Object Perception](https://mlsanity.com/paper/2004.02673/) (2020)
- [Unsupervised Learning of Compositional Scene Representations from Multiple Unspecified Viewpoints](https://mlsanity.com/paper/2112.03568/) (2021)
- [Raven's Progressive Matrices Completion with Latent Gaussian Process Priors](https://mlsanity.com/paper/2103.12045/) (2021)
- [Spatial Mixture Models with Learnable Deep Priors for Perceptual Grouping](https://mlsanity.com/paper/1902.02502/) (2019)
- [Knowledge-Guided Object Discovery with Acquired Deep Impressions](https://mlsanity.com/paper/2103.10611/) (2021)
- [Compositional Law Parsing with Latent Random Functions](https://mlsanity.com/paper/2209.09115/) (2022)

## Built on by (most cited first, 3 of 3)

- [Unsupervised Object-Centric Learning from Multiple Unspecified Viewpoints](https://mlsanity.com/paper/2401.01922/) (2024)
- [Towards Generative Abstract Reasoning: Completing Raven's Progressive Matrix via Rule Abstraction and Selection](https://mlsanity.com/paper/2401.09966/) (2024)
- [Improving Viewpoint-Independent Object-Centric Representations through Active Viewpoint Selection](https://mlsanity.com/paper/2411.00402/) (2024)

## Abstract (arXiv)

When perceiving the world from multiple viewpoints, humans have the ability to reason about the complete objects in a compositional manner even when an object is completely occluded from certain viewpoints. Meanwhile, humans are able to imagine novel views after observing multiple viewpoints. Recent remarkable advances in multi-view object-centric learning still leaves some unresolved problems: 1) The shapes of partially or completely occluded objects can not be well reconstructed. 2) The novel viewpoint prediction depends on expensive viewpoint annotations rather than implicit rules in view representations. In this paper, we introduce a time-conditioned generative model for videos. To reconstruct the complete shape of an object accurately, we enhance the disentanglement between the latent representations of objects and views, where the latent representations of time-conditioned views are jointly inferred with a Transformer and then are input to a sequential extension of Slot Attention to learn object-centric representations. In addition, Gaussian processes are employed as priors of view latent variables for video generation and novel-view prediction without viewpoint annotations. Experiments on multiple datasets demonstrate that the proposed model can make object-centric video decomposition, reconstruct the complete shapes of occluded objects, and make novel-view predictions.