mlsanityData to 7 Oct 2026

RoboDreamer: Learning Compositional World Models for Robot Imagination

arXiv 2404.12377 ↗First posted 18 Apr 2024cs.ROcited by 187

On the map: World models → Embodied and robot world models

Abstract (arXiv)

Text-to-video models have demonstrated substantial potential in robotic decision-making, enabling the imagination of realistic plans of future actions as well as accurate environment simulation. However, one major issue in such models is generalization -- models are limited to synthesizing videos subject to language instructions similar to those seen at training time. This is heavily limiting in decision-making, where we seek a powerful world model to synthesize plans of unseen combinations of objects and actions in order to solve previously unseen tasks in new environments. To resolve this issue, we introduce RoboDreamer, an innovative approach for learning a compositional world model by factorizing the video generation. We leverage the natural compositionality of language to parse instructions into a set of lower-level primitives, which we condition a set of models on to generate videos. We illustrate how this factorization naturally enables compositional generalization, by allowing us to formulate a new natural language instruction as a combination of previously seen components. We further show how such a factorization enables us to add additional multimodal goals, allowing us to specify a video we wish to generate given both natural language instructions and a goal image. Our approach can successfully synthesize video plans on unseen goals in the RT-X, enables successful robot execution in simulation, and substantially outperforms monolithic baseline approaches to video generation.

Key facts. Cited by 187 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 11 of its 36 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

11 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 10

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  4. 2023Learning Interactive Real-World Simulatorscited by 440
  5. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  6. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  7. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  8. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 49
  9. 2023Adaptive Online Replanning with Diffusion Modelscited by 33
  10. 2023Probabilistic Adaptation of Text-to-Video Modelscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 73; 2026: 102.

187 papers.

Sort:

By area:

  1. 2026World Action Models are Zero-shot Policiescited by 411
  2. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 311
  3. 2026Causal World Modeling for Robot Controlcited by 289
  4. 2025Motus: A Unified Latent Action World Modelcited by 231
  5. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  6. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  7. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  8. 2026GigaWorld-Policy: An Efficient Action-Centered World--Action Modelcited by 129
  9. 2025Unified Vision-Language-Action Modelcited by 128
  10. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  11. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  12. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
38 more
  1. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  2. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  3. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
  4. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  5. 20253D and 4D World Modeling: A Surveycited by 64
  6. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulatorscited by 61
  7. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 60
  8. 2025Vidar: Embodied Video Diffusion Model for Generalist Manipulationcited by 58
  9. 2026Motubrain: An Advanced World Action Model for Robot Controlcited by 54
  10. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  11. 2025MindJourney: Test-Time Scaling with World Models for Spatial Reasoningcited by 49
  12. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  13. 2024ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformercited by 45
  14. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  15. 2025EWMBench: Evaluating Scene, Motion, and Semantic Quality in Embodied World Modelscited by 45
  16. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 43
  17. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  18. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  19. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  20. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  21. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  22. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36
  23. 2024AVID: Adapting Video Diffusion Models to World Modelscited by 35
  24. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  25. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  26. 2026Rethinking Video Generation Model for the Embodied Worldcited by 30
  27. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  28. 2025Learning 3D Persistent Embodied World Modelscited by 29
  29. 2025EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modelingcited by 29
  30. 2024EVA: An Embodied World Model for Future Video Anticipationcited by 28
  31. 2025GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulationcited by 26
  32. 2024Grounding Video Models to Actions through Goal Conditioned Explorationcited by 25
  33. 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 25
  34. 2026QuantVLA: Scale-Calibrated Post-Training Quantization for Vision-Language-Action Modelscited by 25
  35. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 25
  36. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  37. 2025A Survey of Interactive Generative Videocited by 23
  38. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23

50 most cited of 187 All 187 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Large Video Planner Enables Generalizable Robot Control
  2. 2026One from Infinity: Actualizing Futures from Pretrained World Models into Robot Actions
  3. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangement
  4. 2026JEPA-WAM: Connecting Generated Visual Instructions to World Action Models through JEPA Latent Representations
  5. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
  6. 2025Image Generation as a Visual Planner for Robotic Manipulation
  7. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  8. 2026InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization