# Object-centric Forward Modeling for Model Predictive Control

arXiv: https://arxiv.org/abs/1910.03568 · HTML: https://mlsanity.com/paper/1910.03568/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2019.

## Built on by (most cited first, 19 of 19)

- [Graph Inverse Reinforcement Learning from Diverse Videos](https://mlsanity.com/paper/2207.14299/) (2022)
- [Learning Physical Dynamics with Subequivariant Graph Neural Networks](https://mlsanity.com/paper/2210.06876/) (2022)
- [VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation](https://mlsanity.com/paper/2407.09829/) (2024)
- [Is an Object-Centric Video Representation Beneficial for Transfer?](https://mlsanity.com/paper/2207.10075/) (2022)
- [Efficient and Interpretable Robot Manipulation with Graph Neural Networks](https://mlsanity.com/paper/2102.13177/) (2021)
- [Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.03319/) (2021)
- [Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Models](https://mlsanity.com/paper/2404.13474/) (2024)
- [3D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenes](https://mlsanity.com/paper/2304.11470/) (2023)
- [Factored World Models for Zero-Shot Generalization in Robotic Manipulation](https://mlsanity.com/paper/2202.05333/) (2022)
- [Object and Relation Centric Representations for Push Effect Prediction](https://mlsanity.com/paper/2102.02100/) (2021)
- [Object-centric Video Prediction without Annotation](https://mlsanity.com/paper/2105.02799/) (2021)
- [Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction](https://mlsanity.com/paper/2107.05140/) (2021)
- [Perceiving Unseen 3D Objects by Poking the Objects](https://mlsanity.com/paper/2302.13375/) (2023)
- [Vision-Language Model Predictive Control for Manipulation Planning and Trajectory Generation](https://mlsanity.com/paper/2504.05225/) (2025)
- [Efficient and Feasible Robotic Assembly Sequence Planning via Graph Representation Learning](https://mlsanity.com/paper/2303.10135/) (2023)
- [Interleaved Multitask Learning with Energy Modulated Learning Progress](https://mlsanity.com/paper/2504.00707/) (2025)
- [SlotGNN: Unsupervised Discovery of Multi-Object Representations and Visual Dynamics](https://mlsanity.com/paper/2310.04617/) (2023)
- [KINet: Unsupervised Forward Models for Robotic Pushing Manipulation](https://mlsanity.com/paper/2202.09006/) (2022)
- [Learning Physical Dynamics for Object-centric Visual Prediction](https://mlsanity.com/paper/2403.10079/) (2024)

## Abstract (arXiv)

We present an approach to learn an object-centric forward model, and show that this allows us to plan for sequences of actions to achieve distant desired goals. We propose to model a scene as a collection of objects, each with an explicit spatial location and implicit visual feature, and learn to model the effects of actions using random interaction data. Our model allows capturing the robot-object and object-object interactions, and leads to more sample-efficient and accurate predictions. We show that this learned model can be leveraged to search for action sequences that lead to desired goal configurations, and that in conjunction with a learned correction module, this allows for robust closed loop execution. We present experiments both in simulation and the real world, and show that our approach improves over alternate implicit or pixel-space forward models. Please see our project page (https://judyye.github.io/ocmpc/) for result videos.