# WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment

arXiv: https://arxiv.org/abs/2402.12275 · HTML: https://mlsanity.com/paper/2402.12275/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 40 of its 62 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 40)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Large Language Models as Commonsense Knowledge for Large-Scale Task Planning](https://mlsanity.com/paper/2305.14078/) (2023)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning](https://mlsanity.com/paper/2305.14909/) (2023)
- [Is Self-Repair a Silver Bullet for Code Generation?](https://mlsanity.com/paper/2306.09896/) (2023)
- [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning](https://mlsanity.com/paper/2006.08381/) (2020)
- [Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning](https://mlsanity.com/paper/2311.17842/) (2023)
- [Language Models Meet World Models: Embodied Experiences Enhance Language Models](https://mlsanity.com/paper/2305.10626/) (2023)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling](https://mlsanity.com/paper/2301.12050/) (2023)
- [Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics](https://mlsanity.com/paper/1706.04317/) (2017)
- [Top-Down Synthesis for Library Learning](https://mlsanity.com/paper/2211.16605/) (2022)
- [LILO: Learning Interpretable Libraries by Compressing and Documenting Code](https://mlsanity.com/paper/2310.19791/) (2023)
- [Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiency](https://mlsanity.com/paper/2309.17382/) (2023)

## Built on by (most cited first, 30 of 107)

- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [Establishing Best Practices for Building Rigorous Agentic Benchmarks](https://mlsanity.com/paper/2507.02825/) (2025)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models](https://mlsanity.com/paper/2503.23350/) (2025)
- [PPTAgent: Generating and Evaluating Presentations Beyond Text-to-Slides](https://mlsanity.com/paper/2501.03936/) (2025)
- [Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning](https://mlsanity.com/paper/2602.10090/) (2026)
- [EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis](https://mlsanity.com/paper/2601.05808/) (2026)
- [Combining Induction and Transduction for Abstract Reasoning](https://mlsanity.com/paper/2411.02272/) (2024)
- [Can Language Models Serve as Text-Based World Simulators?](https://mlsanity.com/paper/2406.06485/) (2024)
- [Code Repair with LLMs gives an Exploration-Exploitation Tradeoff](https://mlsanity.com/paper/2405.17503/) (2024)
- [Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search](https://mlsanity.com/paper/2405.15383/) (2024)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning](https://mlsanity.com/paper/2410.23156/) (2024)
- [Planning with Reasoning using Vision Language World Model](https://mlsanity.com/paper/2509.02722/) (2025)
- [LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines](https://mlsanity.com/paper/2411.08027/) (2024)
- [WALL-E: World Alignment by Rule Learning Improves World Model-based LLM Agents](https://mlsanity.com/paper/2410.07484/) (2024)
- [Text2World: Benchmarking Large Language Models for Symbolic World Model Generation](https://mlsanity.com/paper/2502.13092/) (2025)
- [PoE-World: Compositional World Modeling with Products of Programmatic Experts](https://mlsanity.com/paper/2505.10819/) (2025)
- [Code World Models for General Game Playing](https://mlsanity.com/paper/2510.04542/) (2025)
- [DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environments](https://mlsanity.com/paper/2503.06047/) (2025)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [debug-gym: A Text-Based Environment for Interactive Debugging](https://mlsanity.com/paper/2503.21557/) (2025)
- [Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization](https://mlsanity.com/paper/2602.23008/) (2026)
- [On the Limit of Language Models as Planning Formalizers](https://mlsanity.com/paper/2412.09879/) (2024)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Goals as Reward-Producing Programs](https://mlsanity.com/paper/2405.13242/) (2024)
- [Synthesizing world models for bilevel planning](https://mlsanity.com/paper/2503.20124/) (2025)
- [Modeling Open-World Cognition as On-Demand Synthesis of Probabilistic Models](https://mlsanity.com/paper/2507.12547/) (2025)
- [SSRL: Self-Search Reinforcement Learning](https://mlsanity.com/paper/2508.10874/) (2025)

## Abstract (arXiv)

We give a model-based agent that builds a Python program representing its knowledge of the world based on its interactions with the environment. The world model tries to explain its interactions, while also being optimistic about what reward it can achieve. We define this optimism as a logical constraint between a program and a planner. We study our agent on gridworlds, and on task planning, finding our approach is more sample-efficient compared to deep RL, more compute-efficient compared to ReAct-style agents, and that it can transfer its knowledge across environments by editing its code.