mlsanityData to 14 Oct 2026

DreamGen: Unlocking Generalization in Robot Learning through Video World Models

arXiv 2505.12705 ↗First posted 19 May 2025cs.RO cs.AI cs.LGcited by 179

On the map: World models → Embodied and robot world models

Abstract (arXiv)

We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams.

Key facts. Cited by 179 papers in the map (data through 14 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 71 of its 76 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

71 of 76 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets3 Apr
  2. 2025AdaWorld: Learning Adaptable World Models with Latent Actions24 Mar
  3. 2024RoboDreamer: Learning Compositional World Models for Robot Imagination18 Apr
  4. 2023Learning to Act without Actions17 Dec
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 66

  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  5. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  6. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  7. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  9. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  10. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  11. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
38 more
  1. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  2. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  3. 2023Learning Interactive Real-World Simulatorscited by 458
  4. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  5. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  6. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  7. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  8. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  9. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  10. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  11. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  12. 2024Latent Action Pretraining from Videoscited by 321
  13. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  14. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  15. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  16. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 223
  17. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  18. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  19. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  20. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 206
  21. 2025Unified Video Action Modelcited by 205
  22. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  23. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 164
  24. 2023Video Language Planningcited by 162
  25. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 159
  26. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  27. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 153
  28. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 116
  29. 2022VideoDex: Learning Dexterity from Internet Videoscited by 106
  30. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 106
  31. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  32. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 87
  33. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  34. 2024EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learningcited by 75
  35. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 71
  36. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 70
  37. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 69
  38. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 66

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 23; 2026: 156.

179 papers.

Sort:

By area:

  1. 2026World Action Models are Zero-shot Policiescited by 465
  2. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 300
  3. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  4. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 165
  5. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 137
  6. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  7. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  8. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  9. 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 76
  10. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 70
  11. 2025GigaWorld-0: World Models as Data Engine to Empower Embodied AIcited by 62
  12. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
38 more
  1. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 47
  2. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  3. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 45
  4. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Modelcited by 44
  5. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 40
  6. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 37
  7. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 36
  8. 2026Rethinking Video Generation Model for the Embodied Worldcited by 33
  9. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 31
  10. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAscited by 30
  11. 2025ViPRA: Video Prediction for Robot Actionscited by 26
  12. 2026GigaBrain-0.5M*: a VLA That Learns From World Model-Based Reinforcement Learningcited by 24
  13. 2026RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulationcited by 19
  14. 2025Dexterous World Modelscited by 18
  15. 2026PlayWorld: Learning Robot World Models from Autonomous Playcited by 18
  16. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 18
  17. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 18
  18. 2025Contrastive Representation Regularization for Vision-Language-Action Modelscited by 17
  19. 2026WALL-WM: Carving World Action Modeling at the Event Jointscited by 17
  20. 2025LikePhys: Evaluating Intuitive Physics Understanding in Video Diffusion Models via Likelihood Preferencecited by 16
  21. 2026MaskWAM: Unifying Mask Prompting and Prediction for World-Action Modelscited by 16
  22. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  23. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
  24. 2025Reliable and Scalable Robot Policy Evaluation with Imperfect Simulatorscited by 14
  25. 2026EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewardscited by 14
  26. 2026Physion-Eval: Evaluating Physical Realism in Generated Video via Human Reasoningcited by 14
  27. 2026Code World Model: Coding Agent as World Braincited by 14
  28. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 13
  29. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 12
  30. 2026World Action Models: A Surveycited by 12
  31. 2025AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesiscited by 11
  32. 2025Cosmos-H-Surgical: Learning Surgical Robot Policies from Videos via World Modelingcited by 11
  33. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 11
  34. 2026Flex-π: A Multi-Stream World-Action Model with Compute Flexibilitycited by 11
  35. 2026Olaf-World: Orienting Latent Actions for Video World Modelingcited by 10
  36. 2026Hi-WM: Human-in-the-World-Model for Scalable Robot Post-Trainingcited by 10
  37. 2026HarmoWAM: Harmonizing Generalizable and Precise Manipulation via Adaptive World Action Modelscited by 10
  38. 2025Watch and Learn: Learning to Use Computers from Online Videoscited by 9

50 most cited of 179 All 179 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2018Learning Real-World Robot Policies by Dreaming
  2. 2026Turning Video Models into Generalist Robot Policies
  3. 2026DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
  4. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations
  5. 2026Being-M0.7: A Latent World-Action Model for Humanoid Robots
  6. 2025MinD: Learning A Dual-System World Model for Real-Time Planning and Implicit Risk Analysis
  7. 2026Grounded-Dreamer: Robot Learning from World Model Synthetic Data with Physical Grounding
  8. 2025VideoVLA: Video Generators Can Be Generalizable Robot Manipulators