# OCK: Unsupervised Dynamic Video Prediction with Object-Centric Kinematics

arXiv: https://arxiv.org/abs/2404.18423 · HTML: https://mlsanity.com/paper/2404.18423/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 52 references that are in the map.
- First posted on arXiv 29 Apr 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations](https://mlsanity.com/paper/1907.13052/) (2019)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos](https://mlsanity.com/paper/2206.07764/) (2022)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos](https://mlsanity.com/paper/2205.14065/) (2022)
- [HoloDiffusion: Training a 3D Diffusion Model using 2D Images](https://mlsanity.com/paper/2303.16509/) (2023)
- [Improving Generative Imagination in Object-Centric World Models](https://mlsanity.com/paper/2010.02054/) (2020)
- [Video Transformers: A Survey](https://mlsanity.com/paper/2201.05991/) (2022)
- [Generative Neurosymbolic Machines](https://mlsanity.com/paper/2010.12152/) (2020)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [Folded Recurrent Neural Networks for Future Video Prediction](https://mlsanity.com/paper/1712.00311/) (2017)
- [Self-supervised Visual Reinforcement Learning with Object-centric Representations](https://mlsanity.com/paper/2011.14381/) (2020)

## Built on by (most cited first, 3 of 3)

- [Kinematics-Driven Gaussian Shape Deformation for Blurry Monocular Dynamic Scenes](https://mlsanity.com/paper/2605.08635/) (2026)
- [Event-Conditioned Diagnostics of Kinematic, Contact, and Object-Permanence Structure in Passive Object-State World Models](https://mlsanity.com/paper/2606.28455/) (2026)
- [Abductive World Modeling via Causal Representation Learning](https://mlsanity.com/paper/2609.36985/) (2026)

## Abstract (arXiv)

Human perception involves decomposing complex multi-object scenes into time-static object appearance (i.e., size, shape, color) and time-varying object motion (i.e., position, velocity, acceleration). For machines to achieve human-like intelligence in real-world interactions, understanding these physical properties of objects is essential, forming the foundation for dynamic video prediction. While recent advancements in object-centric transformers have demonstrated potential in video prediction, they primarily focus on object appearance, often overlooking motion dynamics, which is crucial for modeling dynamic interactions and maintaining temporal consistency in complex environments. To address these limitations, we propose OCK, a dynamic video prediction model leveraging object-centric kinematics and object slots. We introduce a novel component named Object Kinematics that comprises explicit object motions, serving as an additional attribute beyond conventional appearance features to model dynamic scenes. The Object Kinematics are integrated into various OCK mechanisms, enabling spatiotemporal prediction of complex object interactions over long video sequences. Our model demonstrates superior performance in handling complex scenes with intricate object attributes and motions, highlighting its potential for applicability in vision-related dynamics learning tasks.