# TextOCVP: Object-Centric Video Prediction with Language Guidance

arXiv: https://arxiv.org/abs/2502.11655 · HTML: https://mlsanity.com/paper/2502.11655/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 67 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors](https://mlsanity.com/paper/2310.12190/) (2023)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [On the Binding Problem in Artificial Neural Networks](https://mlsanity.com/paper/2012.05208/) (2020)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition](https://mlsanity.com/paper/2001.02407/) (2020)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)

## Built on by (most cited first, 1 of 1)

- [SlotDiT: Object-Centric Representations for Diffusion Transformers](https://mlsanity.com/paper/2609.17414/) (2026)

## Abstract (arXiv)

Understanding and forecasting future scene states is critical for autonomous agents to plan and act effectively in complex environments. Object-centric models, with structured latent spaces, have shown promise in modeling object dynamics and predicting future scene states, but often struggle to scale beyond simple synthetic datasets and to integrate external guidance, limiting their applicability in robotics. To address these limitations, we propose TextOCVP, an object-centric model for video prediction guided by textual descriptions. TextOCVP parses an observed scene into object representations, called slots, and utilizes a text-conditioned transformer predictor to forecast future object states and video frames. Our approach jointly models object dynamics and interactions while incorporating textual guidance, enabling accurate and controllable predictions. TextOCVP's structured latent space offers a more precise control of the forecasting process, outperforming several video prediction baselines on two datasets. Additionally, we show that structured object-centric representations provide superior robustness to novel scene configurations, as well as improved controllability and interpretability, enabling more precise and understandable predictions. Videos and code are available at https://play-slot.github.io/TextOCVP.