mlsanityData to 7 Oct 2026

WorldGPT: Empowering LLM as Multimodal World Model

arXiv 2404.18202 ↗First posted 28 Apr 2024cs.AI cs.MMcited by 42

On the map: World models → World models for language agents

Abstract (arXiv)

World models are progressively being employed across diverse fields, extending from basic environment simulation to complex scenario construction. However, existing models are mainly trained on domain-specific states and actions, and confined to single-modality state representations. In this paper, We introduce WorldGPT, a generalist world model built upon Multimodal Large Language Model (MLLM). WorldGPT acquires an understanding of world dynamics through analyzing millions of videos across various domains. To further enhance WorldGPT's capability in specialized scenarios and long-term tasks, we have integrated it with a novel cognitive architecture that combines memory offloading, knowledge retrieval, and context reflection. As for evaluation, we build WorldNet, a multimodal state transition prediction benchmark encompassing varied real-life scenarios. Conducting evaluations on WorldNet directly demonstrates WorldGPT's capability to accurately model state transition patterns, affirming its effectiveness in understanding and predicting the dynamics of complex scenarios. We further explore WorldGPT's emerging potential in serving as a world simulator, helping multimodal agents generalize to unfamiliar domains through efficiently synthesising multimodal instruction instances which are proved to be as reliable as authentic data for fine-tuning purposes. The project is available on urlhttps://github.com/DCDmllm/WorldGPT.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 39 of its 63 references that are in the map. First posted on arXiv 28 Apr 2024.

Builds on

39 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Models18 May

Other references · 38

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2018World Modelscited by 1,444
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2023Mastering Diverse Domains through World Modelscited by 1,193
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
26 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2020Mastering Atari with Discrete World Modelscited by 990
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  5. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  6. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  8. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  9. 2023Learning Interactive Real-World Simulatorscited by 440
  10. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  11. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  12. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  13. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  14. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  15. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  16. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  17. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  18. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  19. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  20. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  21. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  22. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 87
  23. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  24. 2023Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMscited by 35
  25. 2023MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representationscited by 33
  26. 2023Synergistic Integration of Large Language Models and Cognitive Architectures for Robust AI: An Exploratory Analysiscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 21; 2026: 8.

42 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  2. 2024CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLMcited by 74
  3. 20253D and 4D World Modeling: A Surveycited by 64
  4. 2024Auto-Encoding Morph-Tokens for Multimodal LLMcited by 30
  5. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  6. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 24
  7. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  8. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  9. 2025MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Meshcited by 18
  10. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  11. 2025DriVerse: Navigation World Model for Driving Simulation via Multimodal Trajectory Prompting and Motion Alignmentcited by 15
  12. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 14
30 more
  1. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  2. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  3. 2025Guiding Audio Editing with Audio Language Modelcited by 11
  4. 2024Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspectivecited by 10
  5. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  6. 2025Four Principles for Physically Interpretable World Modelscited by 10
  7. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Programcited by 10
  8. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 10
  9. 2025LASER: Lip Landmark Assisted Speaker Detection for Robustnesscited by 6
  10. 2025WALL-E 2.0: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agentscited by 5
  11. 2025CogDriver: Integrating Cognitive Inertia for Temporally Coherent Planning in Autonomous Drivingcited by 5
  12. 2025From 2D to 3D Cognition: A Brief Survey of General World Modelscited by 4
  13. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  14. 2026Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignmentcited by 4
  15. 2026From Pixels to States: Rethinking Interactive World Models as Game Enginescited by 4
  16. 2024ACDC: Autoregressive Coherent Multimodal Generation using Diffusion Correctioncited by 3
  17. 2025Edge General Intelligence Through World Models and Agentic AI: Fundamentals, Solutions, and Challengescited by 3
  18. 2026A Definition and Roadmap for World Modelscited by 3
  19. 2025Large Language models for Time Series Analysis: Techniques, Applications, and Challengescited by 2
  20. 2025Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisonscited by 2
  21. 2026An Empirical Study of World Model Quantizationcited by 2
  22. 2024A Survey for Deep Reinforcement Learning Based Network Intrusion Detectioncited by 1
  23. 2024ARCON: Advancing Auto-Regressive Continuation for Driving Videoscited by 1
  24. 2025A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environmentscited by 1
  25. 2025Short-video Propagation Influence Rating: A New Real-world Dataset and A New Large Graph Modelcited by 1
  26. 2025OptAgent: Optimizing Query Rewriting for E-commerce via Multi-Agent Simulationcited by 0
  27. 2026SiMing-Bench: Evaluating Procedural Correctness from Continuous Interactions in Clinical Skill Videoscited by 0
  28. 2026Model-Based Proactive Cost Generation for Learning Safe Policies Offline with Limited Violation Datacited by 0
  29. 2026Textual Belief States for World Models: Identifiable Representation Learning Under Strict Mediationcited by 0
  30. 2026The CRISTAL Method: Neurosymbolic analysis from AI-synthesized world modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Pandora: Towards General World Model with Natural Language Actions and Video States
  2. 2024From Efficient Multimodal Models to World Models: A Survey
  3. 2026The Trinity of Consistency as a Defining Principle for General World Models
  4. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos
  5. 2026Qwen-AgentWorld: Language World Models for General Agents
  6. 2026WorldAgen: Unified State-Action Prediction with Test-Time World Model Training
  7. 2026World Models Meet Language Models: On the Complementarity of Concrete and Abstract Reasoning
  8. 2026OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining