mlsanityData to 7 Oct 2026

Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets

arXiv 2504.02792 ↗First posted 3 Apr 2025cs.RO cs.AI cs.LGcited by 218

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

Imitation learning has emerged as a promising approach towards building generalist robots. However, scaling imitation learning for large robot foundation models remains challenging due to its reliance on high-quality expert demonstrations. Meanwhile, large amounts of video data depicting a wide range of environments and diverse behaviors are readily available. This data provides a rich source of information about real-world dynamics and agent-environment interactions. Leveraging this data directly for imitation learning, however, has proven difficult due to the lack of action annotation. In this work, we present Unified World Models (UWM), a framework that allows for leveraging both video and action data for policy learning. Specifically, a UWM integrates an action diffusion process and a video diffusion process within a unified transformer architecture, where independent diffusion timesteps govern each modality. By controlling each diffusion timestep, UWM can flexibly represent a policy, a forward dynamics, an inverse dynamics, and a video generator. Through simulated and real-world experiments, we show that: (1) UWM enables effective pretraining on large-scale multitask robot datasets with both dynamics and action predictions, resulting in more generalizable and robust policies than imitation learning, (2) UWM naturally facilitates learning from action-free video data through independent control of modality-specific diffusion timesteps, further improving the performance of finetuned policies. Our results suggest that UWM offers a promising step toward harnessing large, heterogeneous datasets for scalable robot learning, and provides a simple unification between the often disparate paradigms of imitation learning and world modeling. Videos and code are available at https://weirdlabuw.github.io/uwm/.

Key facts. Cited by 218 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 38 of its 48 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

38 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Models24 May
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar

Other references · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2022Flow Matching for Generative Modelingcited by 5,426
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  11. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
24 more
  1. 2022Video Diffusion Modelscited by 2,280
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  4. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  5. 2023Vision Transformers Need Registerscited by 844
  6. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  8. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  9. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  10. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  11. 2024Latent Action Pretraining from Videoscited by 302
  12. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  13. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  14. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  15. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  16. 2023One Transformer Fits All Distributions in Multi-Modal Diffusion at Scalecited by 185
  17. 2024Behavior Generation with Latent Actionscited by 159
  18. 2024ALOHA Unleashed: A Simple Recipe for Robot Dexteritycited by 128
  19. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  20. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  21. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  22. 2024The Ingredients for Robotic Diffusion Transformerscited by 72
  23. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  24. 2024SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spacescited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 34; 2026: 184.

218 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2026World Action Models are Zero-shot Policiescited by 411
  3. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 311
  4. 2026Causal World Modeling for Robot Controlcited by 289
  5. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 274
  6. 2025Motus: A Unified Latent Action World Modelcited by 231
  7. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  8. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  9. 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 149
  10. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  11. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 117
  12. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
38 more
  1. 2025F1: A Vision-Language-Action Model Bridging Understanding and Generation to Actionscited by 85
  2. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 68
  3. 2025Vid2World: Crafting Video Diffusion Models to Interactive World Modelscited by 67
  4. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  5. 2026Unified 4D World Action Modeling from Video Priors with Asynchronous Denoisingcited by 65
  6. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  7. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 45
  8. 2025DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoningcited by 44
  9. 2026World Model for Robot Learning: A Comprehensive Surveycited by 42
  10. 2026RISE: Self-Improving Robot Policy with Compositional World Modelcited by 39
  11. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 36
  12. 2026Do World Action Models Generalize Better than VLAs? A Robustness Studycited by 35
  13. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 33
  14. 2026ImageWAM: Do World Action Models Really Need Video Generation, or Just Image Editing?cited by 33
  15. 2026Rethinking Video Generation Model for the Embodied Worldcited by 30
  16. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  17. 2026ABot-M0.5: Unified Mobility-and-Manipulation World Action Modelcited by 28
  18. 2026OA-WAM: Object-Addressable World Action Model for Robust Robot Manipulationcited by 25
  19. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  20. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 23
  21. 2026τ0-WM: A Unified Video-Action World Model for Robotic Manipulationcited by 21
  22. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  23. 2026FRAPPE: Infusing World Modeling into Generalist Policies via Multiple Future Representation Alignmentcited by 19
  24. 2026Light-WAM: Efficient World Action Models with State-Fusion Action Decodingcited by 19
  25. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  26. 2026When to Trust Imagination: Adaptive Action Execution for World Action Modelscited by 18
  27. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  28. 2026Dream-Tac: A Unified Tactile World Action Model for Contact-Rich Robot Manipulationcited by 17
  29. 2026Efficient-WAM: A 1B-Parameter World-Action Model with Low-Cost Future Imaginationcited by 17
  30. 2026EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Datacited by 17
  31. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  32. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  33. 2025Goal-VLA: Image-Generative VLMs as Object-Centric World Models Empowering Zero-shot Robot Manipulationcited by 15
  34. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  35. 2025Dexterous World Modelscited by 15
  36. 2026Mask World Model: Predicting What Matters for Robust Robot Policy Learningcited by 15
  37. 2026MaskWAM: Unifying Mask Prompting and Prediction for World-Action Modelscited by 15
  38. 2026VTAM: Video-Tactile-Action Models for Complex Physical Interaction Beyond VLAscited by 14

50 most cited of 218 All 218 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
  2. 2026InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data
  3. 2026Interactive World Simulator for Robot Policy Training and Evaluation
  4. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulation
  5. 2026OpenWAM: An Open Framework for Composable World-Action Models
  6. 2026XPACE: Joint World and Action Modeling from Heterogeneous Experience
  7. 2025Action-Free Reasoning for Policy Generalization
  8. 2025Latent Action Pretraining Through World Modeling