mlsanityData to 14 Oct 2026

Object-Centric World Model for Language-Guided Manipulation

arXiv 2503.06170 ↗First posted 8 Mar 2025cs.AI cs.CV cs.ROcited by 16

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

A world model is essential for an agent to predict the future and plan in domains such as autonomous driving and robotics. To achieve this, recent advancements have focused on video generation, which has gained significant attention due to the impressive success of diffusion models. However, these models require substantial computational resources. To address these challenges, we propose a world model leveraging object-centric representation space using slot attention, guided by language instructions. Our model perceives the current state as an object-centric representation and predicts future states in this representation space conditioned on natural language instructions. This approach results in a more compact and computationally efficient model compared to diffusion-based generative alternatives. Furthermore, it flexibly predicts future states based on language instructions, and offers a significant advantage in manipulation tasks where object recognition is crucial. In this paper, we demonstrate that our latent predictive world model surpasses generative world models in visuo-linguo-motor control tasks, achieving superior sample and computation efficiency. We also investigate the generalization performance of the proposed method and explore various strategies for predicting actions using object-centric representations.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 56 of its 56 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

56 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024RoboDreamer: Learning Compositional World Models for Robot Imagination18 Apr
  2. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models16 Oct
  3. 2022Masked World Models for Visual Control28 Jun

Other references · 53

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  9. 2023Mastering Diverse Domains through World Modelscited by 1,253
  10. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  11. 2020Mastering Atari with Discrete World Modelscited by 1,007
  12. 2020On Layer Normalization in the Transformer Architecturecited by 922
38 more
  1. 2020Object-Centric Learning with Slot Attentioncited by 876
  2. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  4. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  5. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  6. 2023Learning Interactive Real-World Simulatorscited by 458
  7. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  8. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 360
  9. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  10. 2021Conditional Object-Centric Learning from Videocited by 248
  11. 2017Stochastic Variational Video Predictioncited by 234
  12. 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
  13. 2022Bridging the Gap to Real-World Object-Centric Learningcited by 220
  14. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  15. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 166
  16. 2023Video Language Planningcited by 162
  17. 2022Model-Based Imitation Learning for Urban Drivingcited by 150
  18. 2021Illiterate DALL-E Learns to Composecited by 147
  19. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 147
  20. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  21. 2019Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEscited by 117
  22. 2020Stochastic Latent Residual Video Predictioncited by 108
  23. 2023Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervisioncited by 99
  24. 2021Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentationcited by 88
  25. 2023Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similaritiescited by 78
  26. 2023TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Predictioncited by 56
  27. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  28. 2023Seer: Language Instructed Video Prediction with Latent Diffusion Modelscited by 52
  29. 2023Invariant Slot Attention: Object Discovery with Slot-Centric Reference Framescited by 51
  30. 2023An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learningcited by 49
  31. 2023Self-supervised Object-Centric Learning for Videoscited by 49
  32. 2020Self-supervised Visual Reinforcement Learning with Object-centric Representationscited by 44
  33. 2022Discovering Objects that Can Movecited by 44
  34. 2024Adaptive Slot Attention: Object Discovery with Dynamic Slot Numbercited by 34
  35. 2024Entity-Centric Reinforcement Learning for Object Manipulation from Pixelscited by 27
  36. 2023Object Discovery from Motion-Guided Tokenscited by 25
  37. 2022Visuomotor Control in Multi-Object Scenes Using Object-Aware Representationscited by 21
  38. 2023Guided Slot Attention for Unsupervised Video Object Segmentationcited by 19

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 10.

16 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  2. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 28
  3. 2025Pixel Motion as Universal Representation for Robot Controlcited by 15
  4. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
  5. 2026World-Model-Augmented Web Agents with Action Correctioncited by 9
  6. 2025Pixel Motion Diffusion is What We Need for Robot Controlcited by 7
  7. 2025Robotic VLA Benefits from Joint Learning with Motion Image Diffusioncited by 7
  8. 2026A Definition and Roadmap for World Modelscited by 5
  9. 2026HAIC: Humanoid Agile Object Interaction Control via Dynamics-Aware World Modelcited by 4
  10. 2026From World Models to World Action Models: A Concise Tutorial for Roboticscited by 4
  11. 2026Latent State Design for World Models under Sufficiency Constraintscited by 3
  12. 2026Robots Need More than VLA and World Modelscited by 2
4 more
  1. 2025Disentangled Representation Learning via Modular Compositional Biascited by 1
  2. 2026World Models for Embodied Intelligence: From Plausible to Controllable to Actionablecited by 1
  3. 2025Learning to Grasp Anything by Playing with Random Toyscited by 0
  4. 2026SlotDiT: Object-Centric Representations for Diffusion Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ObjDream: Guiding Vision-Language-Action Models with Object-Level Understanding and Future State Prediction
  2. 2024Representing Positional Information in Generative World Models for Object Manipulation
  3. 2023FOCUS: Object-Centric World Models for Robotics Manipulation
  4. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulation
  5. 2025LaDi-WM: A Latent Diffusion-based World Model for Predictive Manipulation
  6. 2026World Models for Robotic Manipulation: A Survey
  7. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation
  8. 2026AWM-VLA: AlignedWorld Modeling for Efficient and Explainable Vision-Language-Action Policies