mlsanityData to 14 Oct 2026

WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment

arXiv 2402.12275 ↗First posted 19 Feb 2024cs.AI cs.CLcited by 107

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

We give a model-based agent that builds a Python program representing its knowledge of the world based on its interactions with the environment. The world model tries to explain its interactions, while also being optimistic about what reward it can achieve. We define this optimism as a logical constraint between a program and a planner. We study our agent on gridworlds, and on task planning, finding our approach is more sample-efficient compared to deep RL, more compute-efficient compared to ReAct-style agents, and that it can transfer its knowledge across environments by editing its code.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 40 of its 62 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

40 of 62 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency29 Sep
  2. 2023Thinker: Learning to Plan and Act27 Jul
  3. 2023Reasoning with Language Model is Planning with World Model24 May
  4. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning24 May
  5. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planning23 May
  6. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Models18 May
  7. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling28 Jan

Other references · 33

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  5. 2020Mastering Atari with Discrete World Modelscited by 1,007
  6. 2023Teaching Large Language Models to Self-Debugcited by 988
  7. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  8. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  9. 2017A Tutorial on Thompson Samplingcited by 433
  10. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  11. 2022Transformers are Sample-Efficient World Modelscited by 328
  12. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
21 more
  1. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  2. 2015Contextual Markov Decision Processescited by 220
  3. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  4. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 196
  5. 2018Programmatically Interpretable Reinforcement Learningcited by 193
  6. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
  7. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  8. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 120
  9. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  10. 2017Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physicscited by 58
  11. 2022Top-Down Synthesis for Library Learningcited by 58
  12. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 49
  13. 2021Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planningcited by 43
  14. 2023Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thoughtcited by 39
  15. 2023Interactive Planning Using Large Language Models for Partially Observable Robotics Taskscited by 27
  16. 2023Human-like Few-Shot Learning via Bayesian Reasoning over Natural Languagecited by 22
  17. 2022From Perception to Programs: Regularize, Overparameterize, and Amortizecited by 13
  18. 2023Optimize Planning Heuristics to Rank, not to Estimate Cost-to-Goalcited by 10
  19. 2021A Novel Automated Curriculum Strategy to Solve Hard Sokoban Planning Instancescited by 9
  20. 2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
  21. 2022Left Heavy Tails and the Effectiveness of the Policy and Value Networks in DNN-based best-first search for Sokoban Planningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 46; 2026: 46.

107 papers.

Sort:

By area:

  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  2. 2025Agentic Large Language Models, a surveycited by 108
  3. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  4. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  5. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  6. 2025PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slidescited by 86
  7. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 66
  8. 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 63
  9. 2024Combining Induction and Transduction for Abstract Reasoningcited by 53
  10. 2024Can Language Models Serve as Text-Based World Simulators?cited by 52
  11. 2024Code Repair with LLMs gives an Exploration-Exploitation Tradeoffcited by 51
  12. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 40
38 more
  1. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 40
  2. 2024VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planningcited by 39
  3. 2025Planning with Reasoning using Vision Language World Modelcited by 36
  4. 2024LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Enginescited by 30
  5. 2024WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agentscited by 26
  6. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generationcited by 26
  7. 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 26
  8. 2025Code World Models for General Game Playingcited by 26
  9. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 23
  10. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  11. 2025debug-gym: A Text-Based Environment for Interactive Debuggingcited by 19
  12. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 19
  13. 2024On the Limit of Language Models as Planning Formalizerscited by 18
  14. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  15. 2024Goals as Reward-Producing Programscited by 14
  16. 2025Synthesizing world models for bilevel planningcited by 14
  17. 2025Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Modelscited by 14
  18. 2025SSRL: Self-Search Reinforcement Learningcited by 14
  19. 2026Executable World Models for ARC-AGI-3 in the Era of Coding Agentscited by 14
  20. 2026Code World Model: Coding Agent as World Braincited by 14
  21. 2026Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoningcited by 14
  22. 2024Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Searchcited by 13
  23. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  24. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  25. 2025Generating Symbolic World Models via Test-time Scaling of Large Language Modelscited by 12
  26. 2025LLM-Guided Probabilistic Program Induction for POMDP Model Estimationcited by 12
  27. 2025WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-makingcited by 12
  28. 2025QAgent: An LLM-based Multi-Agent System for Autonomous OpenQASM programmingcited by 11
  29. 2025From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Taskcited by 10
  30. 2025One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Explorationcited by 10
  31. 2025ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpuscited by 10
  32. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9
  33. 2025Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedbackcited by 9
  34. 2024InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instructcited by 8
  35. 2025G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibrationcited by 8
  36. 2025Modeling Others' Minds as Codecited by 7
  37. 2026PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environmentscited by 7
  38. 2024From Code to Play: Benchmarking Program Search for Games Using Large Language Modelscited by 5

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026C-World: A Computer Use Agent Environment Creator
  2. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agents
  3. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Models
  4. 2026Qwen-AgentWorld: Language World Models for General Agents
  5. 2026World Models as an Intermediary between Agents and the Real World
  6. 2026Agent-Editing World Model: Rethinking World Modeling for LLM Agents
  7. 2026From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulation
  8. 2025General agents contain world models