# Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactions

arXiv: https://arxiv.org/abs/2302.11850 · HTML: https://mlsanity.com/paper/2302.11850/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 29 references that are in the map.
- First posted on arXiv 23 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [Scaling Autoregressive Video Models](https://mlsanity.com/paper/1906.02634/) (2019)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects](https://mlsanity.com/paper/1806.01794/) (2018)
- [SCALOR: Generative World Models with Scalable Object Representations](https://mlsanity.com/paper/1910.02384/) (2019)
- [Latent Video Transformer](https://mlsanity.com/paper/2006.10704/) (2020)
- [Improving Generative Imagination in Object-Centric World Models](https://mlsanity.com/paper/2010.02054/) (2020)
- [Compositional Video Prediction](https://mlsanity.com/paper/1908.08522/) (2019)
- [Generative Video Transformer: Can Objects be the Words?](https://mlsanity.com/paper/2107.09240/) (2021)
- [Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers](https://mlsanity.com/paper/1809.03137/) (2018)
- [Unsupervised Object-Based Transition Models for 3D Partially Observable Environments](https://mlsanity.com/paper/2103.04693/) (2021)
- [Segmenting the Future](https://mlsanity.com/paper/1904.10666/) (2019)
- [MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks](https://mlsanity.com/paper/2203.09303/) (2022)
- [Local Frequency Domain Transformer Networks for Video Prediction](https://mlsanity.com/paper/2105.04637/) (2021)
- [Unsupervised Image Decomposition with Phase-Correlation Networks](https://mlsanity.com/paper/2110.03473/) (2021)

## Built on by (most cited first, 19 of 19)

- [Causal-JEPA: Learning World Models through Object-Level Latent Masking](https://mlsanity.com/paper/2602.11389/) (2026)
- [DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particles](https://mlsanity.com/paper/2306.05957/) (2023)
- [Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling](https://mlsanity.com/paper/2603.04553/) (2026)
- [PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning](https://mlsanity.com/paper/2502.07600/) (2025)
- [SLCF-Net: Sequential LiDAR-Camera Fusion for Semantic Scene Completion using a 3D Recurrent U-Net](https://mlsanity.com/paper/2403.08885/) (2024)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [Reasoning-Enhanced Object-Centric Learning for Videos](https://mlsanity.com/paper/2403.15245/) (2024)
- [Smoothing Slot Attention Iterations and Recurrences](https://mlsanity.com/paper/2508.05417/) (2025)
- [OCK: Unsupervised Dynamic Video Prediction with Object-Centric Kinematics](https://mlsanity.com/paper/2404.18423/) (2024)
- [Top-Down Guidance for Learning Object-Centric Representations](https://mlsanity.com/paper/2405.10598/) (2024)
- [Space-Time Forecasting of Dynamic Scenes with Motion-aware Gaussian Grouping](https://mlsanity.com/paper/2602.21668/) (2026)
- [Object-centric proto-symbolic behavioural reasoning from pixels](https://mlsanity.com/paper/2411.17438/) (2024)
- [Rethinking Object-Centric Representations for Video Dynamics Modeling](https://mlsanity.com/paper/2606.23436/) (2026)
- [TextOCVP: Object-Centric Video Prediction with Language Guidance](https://mlsanity.com/paper/2502.11655/) (2025)
- [OC-SOP: Enhancing Vision-Based 3D Semantic Occupancy Prediction by Object-Centric Awareness](https://mlsanity.com/paper/2506.18798/) (2025)
- [VideoPCDNet: Video Parsing and Prediction with Phase Correlation Networks](https://mlsanity.com/paper/2506.19621/) (2025)
- [Event-Conditioned Diagnostics of Kinematic, Contact, and Object-Permanence Structure in Passive Object-State World Models](https://mlsanity.com/paper/2606.28455/) (2026)
- [SlotDiT: Object-Centric Representations for Diffusion Transformers](https://mlsanity.com/paper/2609.17414/) (2026)
- [iSEE: Object Permanence Through Self-Supervision](https://mlsanity.com/paper/2610.01201/) (2026)

## Abstract (arXiv)

We propose a novel framework for the task of object-centric video prediction, i.e., extracting the compositional structure of a video sequence, as well as modeling objects dynamics and interactions from visual observations in order to predict the future object states, from which we can then generate subsequent video frames. With the goal of learning meaningful spatio-temporal object representations and accurately forecasting object states, we propose two novel object-centric video predictor (OCVP) transformer modules, which decouple the processing of temporal dynamics and object interactions, thus presenting an improved prediction performance. In our experiments, we show how our object-centric prediction framework utilizing our OCVP predictors outperforms object-agnostic video prediction models on two different datasets, while maintaining consistent and accurate object representations.