mlsanityData to 13 Oct 2026

DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos

arXiv 2602.06949 ↗First posted 6 Feb 2026cs.RO cs.AI cs.CVcited by 125

On the map: World models → Embodied and robot world models

Abstract (arXiv)

Being able to simulate the outcomes of actions in varied environments will revolutionize the development of generalist agents at scale. However, modeling these world dynamics, especially for dexterous robotics tasks, poses significant challenges due to limited data coverage and scarce action labels. As an endeavor towards this end, we introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos. Our data mixture represents the largest video dataset to date for world model pretraining, spanning a wide range of daily scenarios with diverse objects and skills. To address the scarcity of action labels, we introduce continuous latent actions as unified proxy actions, enhancing interaction knowledge transfer from unlabeled videos. After post-training on small-scale target robot data, DreamDojo demonstrates a strong understanding of physics and precise action controllability. We also devise a distillation pipeline that accelerates DreamDojo to a real-time speed of 10.81 FPS and further improves context consistency. Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning. Systematic evaluation on multiple challenging out-of-distribution (OOD) benchmarks verifies the significance of our method for simulating open-world, contact-rich tasks, paving the way for general-purpose robot world models.

Key facts. Cited by 125 papers in the map (data through 13 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 111 of its 125 references that are in the map. First posted on arXiv 6 Feb 2026.

Builds on

111 of 125 references.

Closest ancestors · 24

Same direction, newest first

  1. 2026Learning Latent Action World Models In The Wild8 Jan
  2. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videos15 Dec
  3. 2025Evaluating Gemini Robotics Policies in a Veo World Simulator11 Dec
  4. 2025Reinforcing Action Policies by Prophesying25 Nov
  5. 2025Scalable Policy Evaluation with Video World Models14 Nov
  6. 2025WMPO: World Model-based Policy Optimization for Vision-Language-Action Models12 Nov
  7. 2025Co-Evolving Latent Action World Models30 Oct
  8. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulation11 Oct
  9. 2025StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representation6 Oct
  10. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators1 Oct
  11. 2025World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Training29 Sep
  12. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation23 Sep
12 more
  1. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models31 Jul
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning11 Jun
  3. 2025WorldEval: World Model as Real-World Robot Policies Evaluator25 May
  4. 2025FLARE: Robot Learning with Implicit World Modeling21 May
  5. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Models19 May
  6. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets3 Apr
  7. 2025AdaWorld: Learning Adaptable World Models with Latent Actions24 Mar
  8. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI17 Oct
  9. 2024IRASim: A Fine-Grained World Model for Robot Manipulation20 Jun
  10. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models24 May
  11. 2023Learning to Act without Actions17 Dec
  12. 2023Structured World Models from Human Videos21 Aug

Other references · 87

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
  3. 2022Scalable Diffusion Models with Transformerscited by 5,973
  4. 2022Classifier-Free Diffusion Guidancecited by 5,785
  5. 2022Flow Matching for Generative Modelingcited by 5,542
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,714
  8. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,610
  9. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,344
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,028
  11. 2023Mastering Diverse Domains through World Modelscited by 1,224
  12. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,197
38 more
  1. 2023One-step Diffusion with Distribution Matching Distillationcited by 991
  2. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 818
  3. 2018Recurrent World Models Facilitate Policy Evolutioncited by 766
  4. 2024Genie: Generative Interactive Environmentscited by 721
  5. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 664
  6. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 640
  7. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 627
  8. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 590
  9. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 485
  10. 2023Learning Interactive Real-World Simulatorscited by 445
  11. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 397
  12. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 397
  13. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 369
  14. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 341
  15. 2024Latent Action Pretraining from Videoscited by 308
  16. 2023Reconstructing Hands in 3D with Transformerscited by 300
  17. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 292
  18. 2024Diffusion Models Are Real-Time Game Enginescited by 286
  19. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  20. 2024Navigation World Modelscited by 264
  21. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  22. 2025Rolling Forcing: Autoregressive Long Video Diffusion in Real Timecited by 215
  23. 2025Video models are zero-shot learners and reasonerscited by 213
  24. 2025Unified Video Action Modelcited by 200
  25. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 198
  26. 2025World Simulation with Video Foundation Models for Physical AIcited by 190
  27. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 180
  28. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 179
  29. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 158
  30. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  31. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  32. 2026Advancing Open-source World Modelscited by 148
  33. 2025Training Agents Inside of Scalable World Modelscited by 131
  34. 2021DriveGAN: Towards a Controllable High-Quality Neural Simulationcited by 125
  35. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 122
  36. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 110
  37. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  38. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 102

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 125.

125 papers.

Sort:

By area:

  1. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 153
  2. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 106
  3. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 94
  4. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 52
  5. 2026LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policiescited by 49
  6. 2026World Action Models: The Next Frontier in Embodied AIcited by 48
  7. 2026World Model for Robot Learning: A Comprehensive Surveycited by 44
  8. 2026AHA-WAM:Asynchronous Horizon-Adaptive World-Action Modeling with Observation-Guided Context Routingcited by 44
  9. 2026Solaris: Building a Multiplayer Video World Model in Minecraftcited by 41
  10. 2026Interactive World Simulator for Robot Policy Training and Evaluationcited by 30
  11. 2026τ0-WM: A Unified Video-Action World Model for Robotic Manipulationcited by 26
  12. 2026VGGRPO: Towards World-Consistent Video Generation with 4D Latent Rewardcited by 22
38 more
  1. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 20
  2. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 17
  3. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  4. 2026Faster-WAM: Efficient Inference-Time Future Conditioning for Robust World Action Modelscited by 15
  5. 2026WorldKV: Efficient World Memory with World Retrieval and Compressioncited by 14
  6. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyondcited by 12
  7. 2026Gamma-World: Generative Multi-Agent World Modeling Beyond Two Playerscited by 12
  8. 2026T-Rex: Tactile-Reactive Dexterous Manipulationcited by 12
  9. 2026World Action Models: A Surveycited by 11
  10. 2026Kinema4D: Kinematic 4D World Modeling for Spatiotemporal Embodied Simulationcited by 10
  11. 2026LARY: A Latent Action Representation Yielding Benchmark for Generalizable Vision-to-Action Alignmentcited by 10
  12. 2026STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulationcited by 10
  13. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 10
  14. 2026Flex-π: A Multi-Stream World-Action Model with Compute Flexibilitycited by 9
  15. 2026World2Act: Latent Action Post-Training from World Model Dynamicscited by 8
  16. 2026MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Modelscited by 8
  17. 2026Masked Visual Actions for Unified World Modelingcited by 8
  18. 2026Motus2: A Self-Evolving General World Model for Dexterous Manipulationcited by 8
  19. 2026OSCAR: Omni-Embodiment Action-Conditioned World Model for Roboticscited by 7
  20. 2026Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learningcited by 6
  21. 2026ETA: A New Agentic Paradigm for Embodied Taskscited by 6
  22. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  23. 2026VAG: Dual-Stream Video-Action Generation for Embodied Data Synthesiscited by 5
  24. 2026Learning Transferable Dynamics Priors from Action to World Modelingcited by 5
  25. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 5
  26. 2026HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editingcited by 5
  27. 2026LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisitioncited by 4
  28. 2026Ambient Diffusion Policy: Imitation Learning from Suboptimal Data in Roboticscited by 4
  29. 2026Human Universal Graspingcited by 4
  30. 2026From World Models to World Action Models: A Concise Tutorial for Roboticscited by 4
  31. 2026RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluationcited by 4
  32. 2026SG-WAM: Self-Guided World Modeling in Geometry-Aware Policy Spacecited by 4
  33. 2026FACT: Failure-Aware Causal Training for World-Action Modelscited by 4
  34. 2026Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Jointscited by 3
  35. 2026EgoExo-WM: Unlocking Exo Video for Ego World Modelscited by 3
  36. 2026Discrete-WAM: Unified Discrete Vision-Action Token Editing for World-Policy Learningcited by 3
  37. 2026AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customizationcited by 3
  38. 2026μ0: A Scalable 3D Interaction-Trace World Modelcited by 3

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026World Action Models are Zero-shot Policies
  2. 2026InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
  3. 2026LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models
  4. 2026PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
  5. 2026GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
  6. 2026JoyAI-RA 0.5: Scaling Robot Manipulation Learning via Dual Action Alignment
  7. 2026DELE-w0.5: Inferring Action from Future Latent State for Robotic Manipulation
  8. 2026Say, Dream, and Act: Learning Video World Models for Instruction-Driven Robot Manipulation