mlsanityData to 7 Oct 2026

WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokens

arXiv 2401.09985 ↗First posted 18 Jan 2024cs.CVcited by 87

On the map: World models → Video-generation world models and simulators

Abstract (arXiv)

World models play a crucial role in understanding and predicting the dynamics of the world, which is essential for video generation. However, existing world models are confined to specific scenarios such as gaming or driving, limiting their ability to capture the complexity of general world dynamic environments. Therefore, we introduce WorldDreamer, a pioneering world model to foster a comprehensive comprehension of general world physics and motions, which significantly enhances the capabilities of video generation. Drawing inspiration from the success of large language models, WorldDreamer frames world modeling as an unsupervised visual sequence modeling challenge. This is achieved by mapping visual inputs to discrete tokens and predicting the masked ones. During this process, we incorporate multi-modal prompts to facilitate interaction within the world model. Our experiments show that WorldDreamer excels in generating videos across different scenarios, including natural scenes and driving environments. WorldDreamer showcases versatility in executing tasks such as text-to-video conversion, image-tovideo synthesis, and video editing. These results underscore WorldDreamer's effectiveness in capturing dynamic elements within diverse general world environments.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 47 of its 53 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

47 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  2. 2022MAGVIT: Masked Generative Video Transformer10 Dec
  3. 2022MaskGIT: Masked Generative Image Transformer8 Feb
  4. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  12. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
31 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  4. 2023Mastering Diverse Domains through World Modelscited by 1,193
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2020Mastering Atari with Discrete World Modelscited by 990
  7. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  9. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  10. 2023ModelScope Text-to-Video Technical Reportcited by 651
  11. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  12. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  13. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  14. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  15. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  16. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  17. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  18. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  19. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  20. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  21. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  22. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  23. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  24. 2022Masked World Models for Visual Controlcited by 169
  25. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  26. 2023DiffiT: Diffusion Vision Transformers for Image Generationcited by 104
  27. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  28. 2022Deep Hierarchical Planning from Pixelscited by 96
  29. 2023Learning to Model the World with Languagecited by 66
  30. 2023On the De-duplication of LAION-2Bcited by 47
  31. 2024aMUSEd: An Open MUSE Reproductioncited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 33; 2026: 28.

87 papers.

Sort:

By area:

  1. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2024DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representationcited by 123
  4. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  5. 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 92
  6. 2024ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restorationcited by 89
  7. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  8. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
  9. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  10. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  11. 2025Stable Video Infinity: Infinite-Length Video Generation with Error Recyclingcited by 68
  12. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
38 more
  1. 2025A Survey of World Models for Autonomous Drivingcited by 60
  2. 2025Long-Context State-Space Video World Modelscited by 59
  3. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  4. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  5. 2024WorldGPT: Empowering LLM as Multimodal World Modelcited by 42
  6. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  7. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  8. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
  9. 2025AI-Generated Video Detection via Perceptual Straighteningcited by 34
  10. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  11. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 30
  12. 2025Astra: General Interactive World Model with Autoregressive Denoisingcited by 30
  13. 2025EmbodieDreamer: Advancing Real2Sim2Real Transfer for Policy Training via Embodied World Modelingcited by 29
  14. 2024Video Diffusion Models: A Surveycited by 28
  15. 2024EVA: An Embodied World Model for Future Video Anticipationcited by 28
  16. 2024SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generationcited by 28
  17. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 28
  18. 2026Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectoriescited by 24
  19. 2025Embodied AI: From LLMs to World Modelscited by 23
  20. 2025FASTer: Toward Efficient Autoregressive Vision Language Action Modeling via Neural Action Tokenizationcited by 21
  21. 2025HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generationcited by 19
  22. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  23. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 19
  24. 2024PIVOT-R: Primitive-Driven Waypoint-Aware World Model for Robotic Manipulationcited by 17
  25. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  26. 2025HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction via Gaussian Restorationcited by 13
  27. 2025World Models for Cognitive Agents: Transforming Edge Intelligence in Future Networkscited by 12
  28. 2026Learning Vision-Language-Action World Models for Autonomous Drivingcited by 11
  29. 2025ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videoscited by 10
  30. 2026GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluationcited by 10
  31. 2026GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearchcited by 10
  32. 2026UniDriveDreamer: A Single-Stage Multimodal World Model for Autonomous Drivingcited by 9
  33. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  34. 2024A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approachescited by 7
  35. 2025AirScape: An Aerial Generative World Model with Motion Controllabilitycited by 7
  36. 2025World Model-Based End-to-End Scene Generation for Accident Anticipation in Autonomous Drivingcited by 7
  37. 2025Discrete Tokenization for Multimodal LLMs: A Comprehensive Surveycited by 7
  38. 2025KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Modelscited by 7

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026WorldMark: A Unified Benchmark Suite for Interactive Video World Models
  2. 2025Can World Models Benefit VLMs for World Dynamics?
  3. 2026World Tokens: Enhancing Embodied Policies with Training-Time World Modeling
  4. 2026DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning
  5. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
  6. 2025WorldVLA: Towards Autoregressive Action World Model
  7. 2026WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory
  8. 2025WorldModelBench: Judging Video Generation Models As World Models