mlsanityData to 7 Oct 2026

Understanding World or Predicting Future? A Comprehensive Survey of World Models

arXiv 2411.14499 ↗First posted 21 Nov 2024cs.CL cs.AI cs.LGcited by 161

On the map: World models → World models for language agents

Abstract (arXiv)

The concept of world models has garnered significant attention due to advancements in multimodal large language models such as GPT-4 and video generation models such as Sora, which are central to the pursuit of artificial general intelligence. This survey offers a comprehensive review of the literature on world models. Generally, world models are regarded as tools for either understanding the present state of the world or predicting its future dynamics. This review presents a systematic categorization of world models, emphasizing two primary functions: (1) constructing internal representations to understand the mechanisms of the world, and (2) predicting future states to simulate and guide decision-making. Initially, we examine the current progress in these two categories. We then explore the application of world models in key domains, including generative games, autonomous driving, robotics, and social simulacra, with a focus on how each domain utilizes these aspects. Finally, we outline key challenges and provide insights into potential future research directions. We summarize the representative papers along with their code repositories in https://github.com/tsinghua-fib-lab/World-Model.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 211 of its 340 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

211 of 340 references.

Closest ancestors · 36

Same direction, newest first

  1. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents10 Nov
  2. 2024Project Sid: Many-agent simulations toward AI civilization31 Oct
  3. 2024SRAP-Agent: Simulating and Optimizing Scarce Resource Allocation Policy with LLM-based Agent18 Oct
  4. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation17 Oct
  5. 2024GenSim: A General Social Simulation Platform with Large Language Model based Agents6 Oct
  6. 2024From Efficient Multimodal Models to World Models: A Survey27 Jun
  7. 2024Agent Planning with World Knowledge Model23 May
  8. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Models15 May
  9. 2024NL2Plan: Robust LLM-Driven Planning from Minimal Text Descriptions7 May
  10. 2024WorldGPT: Empowering LLM as Multimodal World Model28 Apr
  11. 2024A Survey on the Memory Mechanism of Large Language Model based Agents21 Apr
  12. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization27 Feb
24 more
  1. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment19 Feb
  2. 2024Network Formation and Dynamics Among Multi-LLMs16 Feb
  3. 2024Large Language Models are Geographically Biased5 Feb
  4. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities16 Nov
  5. 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarification17 Oct
  6. 2023EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activities16 Oct
  7. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View3 Oct
  8. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency29 Sep
  9. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf9 Sep
  10. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agents27 Jul
  11. 2023GPT4GEO: How a Language Model Sees the World's Geography30 May
  12. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning24 May
  13. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planning23 May
  14. 2023Emergent Representations of Program Semantics in Language Models Trained on Programs18 May
  15. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Models18 May
  16. 2023COKE: A Cognitive Knowledge Graph for Machine Theory of Mind9 May
  17. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr
  18. 2023Evaluating Large Language Models in Theory of Mind Tasks4 Feb
  19. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs24 Oct
  20. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul
  21. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge17 Jun
  22. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts13 Dec
  23. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learning8 Oct
  24. 2020Language Models are Few-Shot Learners28 May

Other references · 175

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
38 more
  1. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  3. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  5. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  6. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2018World Modelscited by 1,444
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  13. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  14. 2020Mastering Atari with Discrete World Modelscited by 990
  15. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  16. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  17. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  18. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  19. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  20. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  21. 2024Genie: Generative Interactive Environmentscited by 706
  22. 2019Solving Rubik's Cube with a Robot Handcited by 699
  23. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  24. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  25. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  26. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  27. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  28. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  29. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  30. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  31. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  32. 2022TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformerscited by 464
  33. 2023Learning Interactive Real-World Simulatorscited by 440
  34. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  35. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  36. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  37. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  38. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380

50 most cited of 175 All 175 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 65; 2026: 95.

161 papers.

Sort:

By area:

  1. 2025Epona: Autoregressive Diffusion World Model for Autonomous Drivingcited by 115
  2. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  3. 2026WorldArena: A Unified Benchmark for Evaluating Perception and Functional Utility of Embodied World Modelscited by 68
  4. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  5. 20253D and 4D World Modeling: A Surveycited by 64
  6. 2025A Survey of World Models for Autonomous Drivingcited by 60
  7. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  8. 2025RoboScape: Physics-informed Embodied World Modelcited by 38
  9. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 33
  10. 2025ViMo: A Generative Visual GUI World Model for App Agentscited by 33
  11. 2025Vision-Language-Action Models for Autonomous Driving: Past, Present, and Futurecited by 32
  12. 2025The Role of World Models in Shaping Autonomous Driving: A Comprehensive Surveycited by 29
38 more
  1. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  2. 20253D Scene Generation: A Surveycited by 24
  3. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMscited by 23
  4. 2025Embodied AI: From LLMs to World Modelscited by 23
  5. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  6. 2026WorldCompass: Reinforcement Learning for Long-Horizon World Modelscited by 22
  7. 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 21
  8. 2025Assessing Adaptive World Models in Machines with Novel Gamescited by 20
  9. 2025Test-Time Adaptation for LLM Agents via Environment Interactioncited by 17
  10. 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 17
  11. 2025User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluationcited by 16
  12. 2024Generative Emergent Communication: Large Language Model is a Collective World Modelcited by 15
  13. 2025UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understandingcited by 15
  14. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  15. 2025TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Modelcited by 14
  16. 2026iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Frameworkcited by 14
  17. 2025AgentSociety Challenge: Designing LLM Agents for User Modeling and Recommendation on Web Platformscited by 12
  18. 2025World Models for Cognitive Agents: Transforming Edge Intelligence in Future Networkscited by 12
  19. 2026Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directionscited by 12
  20. 2026SCOPE: Simulating Cross-game Operations in Playable Environments for FPS World Modelscited by 12
  21. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 11
  22. 2025Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresightcited by 11
  23. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Testcited by 11
  24. 2026VideoWorld 2: Learning Transferable Knowledge from Real-world Videoscited by 11
  25. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10
  26. 2026World of Workflows: A Benchmark for Bringing World Models to Enterprise Systemscited by 10
  27. 2026ExploreVLA: Dense World Modeling and Exploration for End-to-End Autonomous Drivingcited by 10
  28. 2025Beyond Static Responses: Multi-Agent LLM Systems as a New Paradigm for Social Science Researchcited by 9
  29. 2025What Happens Next? Anticipating Future Motion by Generating Point Trajectoriescited by 9
  30. 2026Language-Conditioned World Modeling for Visual Navigationcited by 9
  31. 2026PhysInOne: Visual Physics Learning and Reasoning in One Suitecited by 9
  32. 2025G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibrationcited by 8
  33. 2026WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platformcited by 8
  34. 2025Simulating Before Planning: Constructing Intrinsic User World Model for User-Tailored Dialogue Policy Planningcited by 7
  35. 2025AirScape: An Aerial Generative World Model with Motion Controllabilitycited by 7
  36. 2025KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Modelscited by 7
  37. 2025LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoEcited by 7
  38. 2026InSpatio-WorldFM: An Open-Source Real-Time Generative Frame Modelcited by 7

50 most cited of 161 All 161 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications
  2. 2026A Tutorial on World Models and Physical AI
  3. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Tasks
  4. 2026Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling
  5. 2026From World Models to World Action Models: A Concise Tutorial for Robotics
  6. 2025Trends in Motion Prediction Toward Deployable and Generalizable Autonomy: A Revisit and Perspectives
  7. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulation
  8. 2025Critique of World Model