mlsanityData to 14 Oct 2026

Evaluating the World Model Implicit in a Generative Model

arXiv 2406.03689 ↗First posted 6 Jun 2024cs.CL cs.AIcited by 113

On the map: World models → World models for language agents

Abstract (arXiv)

Recent work suggests that large language models may implicitly learn world models. How should we assess this possibility? We formalize this question for the case where the underlying reality is governed by a deterministic finite automaton. This includes problems as diverse as simple logical reasoning, geographic navigation, game-playing, and chemistry. We propose new evaluation metrics for world model recovery inspired by the classic Myhill-Nerode theorem from language theory. We illustrate their utility in three domains: game playing, logic puzzles, and navigation. In all domains, the generative models we consider do well on existing diagnostics for assessing world models, but our evaluation metrics reveal their world models to be far less coherent than they appear. Such incoherence creates fragility: using a generative model to solve related but subtly different tasks can lead to failures. Building generative models that meaningfully capture the underlying logic of the domains they model would be immensely valuable; our results suggest new ways to assess how close a given model is to that goal.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 26 of its 41 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

26 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Linear Latent World Models in Simple Transformers: A Case Study on Othello-GPT11 Oct
  2. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning24 May

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2023Mistral 7Bcited by 3,164
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2019Designing and Interpreting Probes with Control Taskscited by 640
  11. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  12. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
12 more
  1. 2022Transformers Learn Shortcuts to Automatacited by 268
  2. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  3. 2024The Illusion of State in State-Space Modelscited by 187
  4. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  5. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  6. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 119
  7. 2021Chess as a Testbed for Language Model State Trackingcited by 72
  8. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  9. 2022Analyzing Generalization of Vision and Language Navigation to Unseen Outdoor Areascited by 24
  10. 2020Generating Landmark Navigation Instructions from Maps as a Graph-to-Text Problemcited by 22
  11. 2018On Evaluating the Generalization of LSTM Models in Formal Languagescited by 17
  12. 2023Large Language Models on the Chessboard: A Study on ChatGPT's Formal Language Comprehension and Complex Reasoning Skillscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 50; 2026: 55.

113 papers.

Sort:

By area:

  1. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  2. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  3. 2024ICLR: In-Context Learning of Representationscited by 50
  4. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 47
  5. 2025(How) Do Language Models Track State?cited by 34
  6. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 31
  7. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generationcited by 26
  8. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 22
  9. 2025Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesiscited by 22
  10. 2025Assessing Adaptive World Models in Machines with Novel Gamescited by 20
  11. 2025Agentic AI Needs a Systems Theorycited by 19
  12. 2025Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Trackingcited by 18
38 more
  1. 2025A Survey on Large Language Models for Automated Planningcited by 17
  2. 2025Potemkin Understanding in Large Language Modelscited by 15
  3. 2024Does ChatGPT Have a Mind?cited by 14
  4. 2024Counterfactual Token Generation in Large Language Modelscited by 14
  5. 2025Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Modelscited by 14
  6. 2025WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-makingcited by 12
  7. 2025From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Taskcited by 10
  8. 2025One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Explorationcited by 10
  9. 2026How Should World Models Be Evaluated for Embodied Decision-Making? A Decision-Making-Centric Positioncited by 10
  10. 2025CORE: Full-Path Evaluation of LLM Agents Beyond Final Statecited by 9
  11. 2025AIA Forecaster: Technical Reportcited by 9
  12. 2025G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibrationcited by 8
  13. 2025What Does it Mean for a Neural Network to Learn a "World Model"?cited by 8
  14. 2025Large Language Models and Emergence: A Complex Systems Perspectivecited by 7
  15. 2026WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agentscited by 7
  16. 2025Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environmentscited by 6
  17. 2025Position: Foundation Models for Tabular Data within Systemic Contexts Need Groundingcited by 6
  18. 2025Linear Spatial World Models Emerge in Large Language Modelscited by 6
  19. 2026Learning State-Tracking from Code Using Linear RNNscited by 6
  20. 2025Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Surveycited by 5
  21. 2026Self-Improving World Modelling with Latent Actionscited by 5
  22. 2026Nano World Models: A Minimalist Implementation of Future Video Predictioncited by 5
  23. 2025Enabling AI Scientists to Recognize Innovation: A Domain-Agnostic Algorithm for Assessing Noveltycited by 4
  24. 2025Tracking World States with Language Models: State-Based Evaluation Using Chesscited by 4
  25. 2026Characterizing the Effect of Noise in Language Generation in the Limitcited by 4
  26. 2024Towards a Science Exocortexcited by 3
  27. 2024A Hierarchical Language Model For Interpretable Graph Reasoningcited by 3
  28. 2025The Limits of Predicting Agents from Behaviourcited by 3
  29. 2025Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibilitycited by 3
  30. 2025Edge General Intelligence Through World Models and Agentic AI: Fundamentals, Solutions, and Challengescited by 3
  31. 2026The Observer Effect in World Models: Invasive Adaptation Corrupts Latent Physicscited by 3
  32. 2026MetaMind: General and Cognitive World Models in Multi-Agent Systems by Meta-Theory of Mindcited by 3
  33. 2026Toward Consistent World Models with Multi-Token Prediction and Latent Semantic Enhancementcited by 3
  34. 2026The Intervention Gap in Latent World Modelscited by 3
  35. 2026Dutch Books for Language Modelscited by 3
  36. 2025Revisiting Rogers' Paradox in the Context of Human-AI Interactioncited by 2
  37. 2025Emergent Stack Representations in Modeling Counter Languages Using Transformerscited by 2
  38. 2025LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?cited by 2

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026An Alternative Trajectory for Generative AI
  2. 2026World Agent: Can Language Models Keep a World Running?
  3. 2024Making Large Language Models into World Models with Precondition and Effect Knowledge
  4. 2025Active Confusion Expression in Large Language Models: Leveraging World Models toward Better Social Reasoning
  5. 2026Why We Need World Models for AGI: Where LLMs Fail and How World Models May Outperform
  6. 2026The Token Games: Evaluating Language Model Reasoning with Puzzle Duels
  7. 2023WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models
  8. 2026Do World Models Learn Global Understanding?