mlsanityData to 14 Oct 2026

Learning to Model the World with Language

arXiv 2308.01399 ↗First posted 31 Jul 2023cs.CL cs.AI cs.LGcited by 68

Not on the map. Why?

Cited by 26 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

To interact with humans and act in the world, agents need to understand the range of language that people use and relate it to the visual world. While current agents can learn to execute simple language instructions, we aim to build agents that leverage diverse language -- language like "this button turns on the TV" or "I put the bowls away" -- that conveys general knowledge, describes the state of the world, provides interactive feedback, and more. Our key idea is that agents should interpret such diverse language as a signal that helps them predict the future: what they will observe, how the world will behave, and which situations will be rewarded. This perspective unifies language understanding with future prediction as a powerful self-supervised learning objective. We instantiate this in Dynalang, an agent that learns a multimodal world model to predict future text and image representations, and learns to act from imagined model rollouts. While current methods that learn language-conditioned policies degrade in performance with more diverse types of language, we show that Dynalang learns to leverage environment descriptions, game rules, and instructions to excel on tasks ranging from game-playing to navigating photorealistic home scans. Finally, we show that our method enables additional capabilities due to learning a generative model: Dynalang can be pretrained on text-only data, enabling learning from offline datasets, and generate language grounded in an environment.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 72 of its 92 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

72 of 92 references.

References · 72

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
38 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2019Language Models as Knowledge Bases?cited by 1,936
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  5. 2023Mastering Diverse Domains through World Modelscited by 1,253
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  10. 2020Mastering Atari with Discrete World Modelscited by 1,007
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  12. 2022A Generalist Agentcited by 808
  13. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  14. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  15. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  16. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  17. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  18. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  19. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  20. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  21. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  22. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  23. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  24. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  25. 2017Embodied Question Answeringcited by 336
  26. 2016Hierarchical Question-Image Co-Attention for Visual Question Answeringcited by 278
  27. 2019Vision-and-Dialog Navigationcited by 250
  28. 2020Experience Grounds Languagecited by 242
  29. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  30. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  31. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  32. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  33. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  34. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
  35. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  36. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  37. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  38. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 169

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 28; 2025: 16; 2026: 14.

68 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  3. 2024Navigation World Modelscited by 272
  4. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 261
  5. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 191
  6. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  7. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  8. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 116
  9. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 108
  10. 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89
  11. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  12. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 56
38 more
  1. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  2. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 48
  3. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  4. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  5. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 40
  6. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  7. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  8. 2024iLLM-TSC: Integration reinforcement learning and large language model for traffic signal control policy improvementcited by 23
  9. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  10. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitationcited by 18
  11. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  12. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 17
  13. 2023LLM-State: Open World State Representation for Long-horizon Task Planning with Large Language Modelcited by 15
  14. 2026Policy and World Modeling Co-Training for Language Agentscited by 15
  15. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  16. 2024Language-Guided World Models: A Model-Based Approach to AI Controlcited by 12
  17. 2026EgoForge: Goal-Directed Egocentric World Simulatorcited by 12
  18. 2025ProphetDWM: A Driving World Model for Rolling Out Future Actions and Videoscited by 11
  19. 2024LLM-Based World Models Can Make Decisions Solely, But Rigorous Evaluations are Neededcited by 10
  20. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  21. 2024Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewardscited by 9
  22. 2025CORE: Full-Path Evaluation of LLM Agents Beyond Final Statecited by 9
  23. 2023LanGWM: Language Grounded World Modelcited by 8
  24. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  25. 2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7
  26. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learningcited by 7
  27. 2025MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXcited by 7
  28. 2024World Models with Hints of Large Language Models for Goal Achievingcited by 6
  29. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  30. 2024MAPLE: A Framework for Active Preference Learning Guided by Large Language Modelscited by 6
  31. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  32. 2023LaGR-SEQ: Language-Guided Reinforcement Learning with Sample-Efficient Queryingcited by 5
  33. 2024Make the Pertinent Salient: Task-Relevant Reconstruction for Visual Control with Distractionscited by 5
  34. 2025Looking beyond the next tokencited by 5
  35. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  36. 2025Learning Actionable World Models for Industrial Process Controlcited by 4
  37. 2024Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Modelscited by 3
  38. 2025Dream to Recall: Imagination-Guided Experience Retrieval for Memory-Persistent Vision-and-Language Navigationcited by 3

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?
  2. 2025Language-conditioned world model improves policy generalization by reading environmental descriptions
  3. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Models
  4. 2026EVOKE: Eliciting World Knowledge in Agents for Transferable Decision-Making
  5. 2024Making Large Language Models into World Models with Precondition and Effect Knowledge
  6. 2024Enhancing Agent Learning through World Dynamics Modeling
  7. 2023Towards A Unified Agent with Foundation Models
  8. 2026Qwen-AgentWorld: Language World Models for General Agents