mlsanityData to 14 Oct 2026

Vid2World: Crafting Video Diffusion Models to Interactive World Models

arXiv 2505.14357 ↗First posted 20 May 2025cs.CV cs.LGcited by 76

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

World models, which predict future transitions from past observation and action sequences, have shown great promise for improving data efficiency in sequential decision-making. However, existing world models often require extensive domain-specific training and still produce low-fidelity, coarse predictions, limiting their usefulness in complex environments. In contrast, video diffusion models trained on large-scale internet data have demonstrated impressive capabilities in generating high-quality videos that capture diverse real-world dynamics. In this work, we present Vid2World, a general approach for leveraging and transferring pre-trained video diffusion models into interactive world models. To bridge the gap, Vid2World systematically explores video diffusion causalization, reshaping both the architecture and training objective of pre-trained models to enable autoregressive generation. Additionally, it incorporates a causal action guidance mechanism to enhance action controllability in the resulting interactive world models. Extensive experiments across multiple domains, including robot manipulation, 3D game simulation, and open-world navigation, demonstrate that our method offers a scalable and effective pathway for repurposing highly capable video diffusion models into interactive world models.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 59 of its 82 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

59 of 82 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft11 Apr
  2. 2025GameFactory: Creating New Games with Generative Interactive Videos14 Jan
  3. 2024AVID: Adapting Video Diffusion Models to World Models1 Oct
  4. 2024Genie: Generative Interactive Environments23 Feb
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  6. 2023Learning Interactive Real-World Simulators9 Oct
  7. 2022Scalable Diffusion Models with Transformers19 Dec
  8. 2021Learning Transferable Visual Models From Natural Language Supervision26 Feb
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  10. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2022Video Diffusion Modelscited by 2,305
37 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  2. 2023Consistency Modelscited by 1,822
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  7. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  8. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  10. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 681
  11. 2024LTX-Video: Realtime Video Latent Diffusioncited by 581
  12. 2025Mean Flows for One-step Generative Modelingcited by 561
  13. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  14. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  15. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  16. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 433
  17. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  18. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 319
  19. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 310
  20. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  21. 2024Navigation World Modelscited by 272
  22. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 240
  23. 2024RoboDreamer: Learning Compositional World Models for Robot Imaginationcited by 195
  24. 2025Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200kcited by 163
  25. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 146
  26. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  27. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  28. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 106
  29. 2025TesserAct: Learning 4D Embodied World Modelscited by 94
  30. 2023Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligencecited by 56
  31. 2024Can Language Models Serve as Text-Based World Simulators?cited by 52
  32. 2021Counter-Strike Deathmatch with Large-Scale Behavioural Cloningcited by 50
  33. 2025Pre-Trained Video Generative Models as World Simulatorscited by 44
  34. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generationcited by 26
  35. 2020Neural Game Engine: Accurate learning of generalizable forward models from pixelscited by 12
  36. 2024Language Agents Meet Causality -- Bridging LLMs and Causal World Modelscited by 9
  37. 2025Trajectory World Models for Heterogeneous Environmentscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 64.

76 papers.

Sort:

By area:

  1. 2026DreamDojo: A Generalist Robot World Model from Large-Scale Human Videoscited by 135
  2. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 96
  3. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  4. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 51
  5. 2026World Model for Robot Learning: A Comprehensive Surveycited by 46
  6. 2025Astra: General Interactive World Model with Autoregressive Denoisingcited by 33
  7. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 30
  8. 2026Qwen-AgentWorld: Language World Models for General Agentscited by 28
  9. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 23
  10. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 22
  11. 2026BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Maskscited by 20
  12. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 18
38 more
  1. 2026Addressable Memory for Video World Modelscited by 15
  2. 2026Code World Model: Coding Agent as World Braincited by 14
  3. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 14
  4. 2025Co-Evolving Latent Action World Modelscited by 13
  5. 2025KlingAvatar 2.0 Technical Reportcited by 13
  6. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 13
  7. 2026Geometry-Aware Implicit Memory for Video World Modelscited by 13
  8. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 11
  9. 2026ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Modelscited by 10
  10. 2026Incantation: Natural Language as the Action Interface for Multi-Entity Video World Modelscited by 10
  11. 2026How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Positioncited by 10
  12. 2026ReWorld: An Interactive World Model with Long-Horizon Memorycited by 10
  13. 2026RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Modelscited by 8
  14. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
  15. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  16. 2025The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Textcited by 7
  17. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 7
  18. 2026A Mechanistic View on Video Generation as World Models: State and Dynamicscited by 6
  19. 2026PhysVid: Physics Aware Local Conditioning for Generative Video Modelscited by 6
  20. 2025PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Modelscited by 5
  21. 2026FashionChameleon: Towards Real-Time and Interactive Human-Garment Video Customizationcited by 5
  22. 2026Nano World Models: A Minimalist Implementation of Future Video Predictioncited by 5
  23. 2026Population-Scalable Multi-Agent World Modelingcited by 5
  24. 2026Latent World Models with Monotone Planning Costs for Image-Goal Navigationcited by 5
  25. 2026DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulationcited by 5
  26. 2025MATRIX: Mask Track Alignment for Interaction-aware Video Generationcited by 4
  27. 2026Diffusion Model as a Generalist Segmentation Learnercited by 4
  28. 2026Learning Visual Feature-Based World Models via Residual Latent Actioncited by 4
  29. 2026RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluationcited by 4
  30. 2026DriftWorld: Fast World Modeling through Driftingcited by 4
  31. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  32. 2026PanoWorld: Geometry-Consistent Panoramic Video World Modelingcited by 3
  33. 2026World Models as Group Actionscited by 3
  34. 2026VGI-Bench: Probing Visual Intelligence in Video Generation Modelscited by 3
  35. 2025VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos?cited by 2
  36. 2026From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulationcited by 2
  37. 2026World-Ego Modeling for Long-Horizon Evolution in Hybrid Embodied Taskscited by 2
  38. 2026Physically Viable World Models: A Case for Query-Conditioned Embodied AIcited by 2

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MiniWorld: Democratizing the Training of Video World Models from Scratch
  2. 2026minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models
  3. 2026Vid2WAM: Distilling Video Diffusion Priors into World Action Models
  4. 2026Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers
  5. 2026Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models
  6. 2026Causal World Modeling for Robot Control
  7. 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Models
  8. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world model