mlsanityData to 14 Oct 2026

Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search

arXiv 2405.15383 ↗First posted 24 May 2024cs.AIcited by 40

On the map: World models → World models for language agents

Abstract (arXiv)

In this work we consider Code World Models, world models generated by a Large Language Model (LLM) in the form of Python code for model-based Reinforcement Learning (RL). Calling code instead of LLMs for planning has potential to be more precise, reliable, interpretable, and extremely efficient. However, writing appropriate Code World Models requires the ability to understand complex instructions, to generate exact code with non-trivial logic and to self-debug a long program with feedback from unit tests and environment trajectories. To address these challenges, we propose Generate, Improve and Fix with Monte Carlo Tree Search (GIF-MCTS), a new code generation strategy for LLMs. To test our approach in an offline RL setting, we introduce the Code World Models Benchmark (CWMB), a suite of program synthesis and planning tasks comprised of 18 diverse RL environments paired with corresponding textual descriptions and curated trajectories. GIF-MCTS surpasses all baselines on the CWMB and two other benchmarks, and we show that the Code World Models synthesized with it can be successfully used for planning, resulting in model-based RL agents with greatly improved sample efficiency and inference speed.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 48 of its 68 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

48 of 68 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment19 Feb
  2. 2024Language-Guided World Models: A Model-Based Approach to AI Control24 Jan
  3. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought22 Jun
  4. 2023Reasoning with Language Model is Planning with World Model24 May
  5. 2023Leveraging Pre-trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning24 May
  6. 2020Language Models are Few-Shot Learners28 May

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  12. 2018World Modelscited by 1,513
30 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  3. 2020Mastering Atari with Discrete World Modelscited by 1,007
  4. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  5. 2021Measuring Coding Challenge Competence With APPScited by 902
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  7. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  8. 2024Genie: Generative Interactive Environmentscited by 747
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  10. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  12. 2023Learning Interactive Real-World Simulatorscited by 458
  13. 2022CodeT: Code Generation with Generated Testscited by 384
  14. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  15. 2022Transformers are Sample-Efficient World Modelscited by 328
  16. 2023Photorealistic Video Generation with Diffusion Modelscited by 306
  17. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  18. 2018Programmatically Interpretable Reinforcement Learningcited by 193
  19. 2023Planning with Large Language Models for Code Generationcited by 182
  20. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  21. 2023Language-Driven Representation Learning for Roboticscited by 165
  22. 2023Learning to Model the World with Languagecited by 68
  23. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 64
  24. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  25. 2021Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planningcited by 43
  26. 2023Learning adaptive planning representations with natural language guidancecited by 36
  27. 2019Few-Shot Bayesian Imitation Learning with Logical Program Policiescited by 30
  28. 2021Grounding Predicates through Actionscited by 25
  29. 2023Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programscited by 20
  30. 2021Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environmentscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 22.

40 papers.

Sort:

By area:

  1. 2025Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Designcited by 109
  2. 2025Text2World: Benchmarking Large Language Models for Symbolic World Model Generationcited by 26
  3. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  4. 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 26
  5. 2025A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworkscited by 21
  6. 2025Navigating the Alpha Jungle: An LLM-Powered MCTS Framework for Formulaic Factor Miningcited by 16
  7. 2025Synthesizing world models for bilevel planningcited by 14
  8. 2026Executable World Models for ARC-AGI-3 in the Era of Coding Agentscited by 14
  9. 2026Code World Model: Coding Agent as World Braincited by 14
  10. 2025Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuningcited by 13
  11. 2026Generative Visual Code Mobile World Modelscited by 13
  12. 2025Generating Symbolic World Models via Test-time Scaling of Large Language Modelscited by 12
28 more
  1. 2026EgoForge: Goal-Directed Egocentric World Simulatorcited by 12
  2. 2025One Life to Learn: Inferring Symbolic World Models for Stochastic Environments from Unguided Explorationcited by 10
  3. 2025Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedbackcited by 9
  4. 2026Budget-Constrained Agentic Large Language Models: Intention-Based Planning for Costly Tool Usecited by 8
  5. 2026Debugging code world modelscited by 5
  6. 2025Adamas: Hadamard Sparse Attention for Efficient Long-Context Inferencecited by 4
  7. 2026Twin: Playing an Unknown Game with a Test-Time Digital Twincited by 4
  8. 2025Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Trainingcited by 3
  9. 2025Zero-Shot Privacy-Aware Text Rewriting via Iterative Tree Searchcited by 3
  10. 2026Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?cited by 3
  11. 2026AI for Games in the Foundation Model Eracited by 3
  12. 2025Streaming Looking Ahead with Token-level Self-rewardcited by 2
  13. 2026Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedbackcited by 2
  14. 2025Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisionscited by 1
  15. 2026Joint Learning of Hierarchical Neural Options and Abstract World Modelcited by 1
  16. 2026COvolve: Adversarial Co-Evolution of Large-Language-Model-Generated Policies and Environments via Two-Player Zero-Sum Gamecited by 1
  17. 2026Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Searchcited by 1
  18. 2026World-Time Compute with Verified Code World Modelscited by 1
  19. 2026EMPIRIC: Experiment-Driven Learning of Residual World Models for Robot Planningcited by 1
  20. 2025GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learningcited by 0
  21. 2025Lessons from Neuroscience for AI: How integrating Actions, Compositional Structure and Episodic Memory could enable Safe, Interpretable and Human-Like AIcited by 0
  22. 2026Code World Models for Parameter Control in Evolutionary Algorithmscited by 0
  23. 2026Hypothesis Generation and Inductive Inference in Children and Language Modelscited by 0
  24. 2026ROSUM-MCTS: Monte Carlo Tree Search-Inspired HDL Code Summarization with Structural Rewardscited by 0
  25. 2026Quo Vadis, World Modeling?cited by 0
  26. 2026When LLM Meets Tree Search: A Systematic View of Inference as Search in Large Language Modelscited by 0
  27. 2026Schema: Discovering Unknown Environments via Agentic Program Inductioncited by 0
  28. 2026D-World: An Explicit World-Language-Action Model with Recursive Self-Improvement under Dual Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Distilling Game Code World Model Generation into Lightweight Large Language Models
  2. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Models
  3. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey
  4. 2024StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
  5. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
  6. 2025Code World Models for General Game Playing
  7. 2023Self-planning Code Generation with Large Language Models
  8. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agents