mlsanityData to 7 Oct 2026

From Masks to Worlds: A Hitchhiker's Guide to World Models

arXiv 2510.20668 ↗First posted 23 Oct 2025cs.LGcited by 5

On the map: World models → World models for language agents

Abstract (arXiv)

This is not a typical survey of world models; it is a guide for those who want to build worlds. We do not aim to catalog every paper that has ever mentioned a ``world model". Instead, we follow one clear road: from early masked models that unified representation learning across modalities, to unified architectures that share a single paradigm, then to interactive generative models that close the action-perception loop, and finally to memory-augmented systems that sustain consistent worlds over time. We bypass loosely related branches to focus on the core: the generative heart, the interactive loop, and the memory system. We show that this is the most promising path towards true world models.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 108 of its 119 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

108 of 119 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr
  2. 2020Language Models are Few-Shot Learners28 May

Other references · 106

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  12. 2024GPT-4o System Cardcited by 4,467
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  11. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  13. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  14. 2018World Modelscited by 1,444
  15. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  16. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  17. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  18. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  19. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  20. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  21. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  22. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  23. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  24. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  25. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  26. 2025Large Language Diffusion Modelscited by 872
  27. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  28. 2022A Generalist Agentcited by 802
  29. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  30. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  31. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  32. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  33. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  34. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  35. 2024Genie: Generative Interactive Environmentscited by 706
  36. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  37. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  38. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025How Far Are Surgeons from Surgical World Models? A Pilot Study on Zero-shot Surgical Video Generation with Expert Assessmentcited by 11
  2. 2025RecTok: Reconstruction Distillation along Rectified Flowcited by 4
  3. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
  4. 2026HelloWorld: Enabling Socially Interactive Characters in Video World Modelscited by 1
  5. 2026Discrete Diffusion Bridges for Spatiotemporally Aligned Image Translation and Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025World-in-World: World Models in a Closed-Loop World
  2. 2026WorldString: Actionable World Representation
  3. 2026OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining
  4. 2026Platonic Representation Hypothesis on World Models
  5. 2026Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling
  6. 2025On Memory: A comparison of memory mechanisms in world models
  7. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks
  8. 2026From Generative Engines to Actionable Simulators: The Imperative of Physical Grounding in World Models