# From Code to Play: Benchmarking Program Search for Games Using Large Language Models

arXiv: https://arxiv.org/abs/2412.04057 · HTML: https://mlsanity.com/paper/2412.04057/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 58 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Programmatically Interpretable Reinforcement Learning](https://mlsanity.com/paper/1804.02477/) (2018)
- [Procedural Content Generation via Machine Learning (PCGML)](https://mlsanity.com/paper/1702.00539/) (2017)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment](https://mlsanity.com/paper/2402.12275/) (2024)
- [LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristics](https://mlsanity.com/paper/2405.20132/) (2024)
- [Few-Shot Bayesian Imitation Learning with Logical Program Policies](https://mlsanity.com/paper/1904.06317/) (2019)
- [Game Generation via Large Language Models](https://mlsanity.com/paper/2404.08706/) (2024)
- [GAVEL: Generating Games Via Evolution and Language Models](https://mlsanity.com/paper/2407.09388/) (2024)
- [Inductive general game playing](https://mlsanity.com/paper/1906.09627/) (2019)
- [Learning MDL logic programs from noisy data](https://mlsanity.com/paper/2308.09393/) (2023)
- [Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Search](https://mlsanity.com/paper/2405.16450/) (2024)
- [Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Sampling](https://mlsanity.com/paper/2402.19471/) (2024)
- [The Ink Splotch Effect: A Case Study on ChatGPT as a Co-Creative Game Designer](https://mlsanity.com/paper/2403.02454/) (2024)
- [Evolving simple programs for playing Atari games](https://mlsanity.com/paper/1806.05695/) (2018)
- [Choosing Well Your Opponents: How to Guide the Synthesis of Programmatic Strategies](https://mlsanity.com/paper/2307.04893/) (2023)
- [Learning of Generalizable and Interpretable Knowledge in Grid-Based Reinforcement Learning Environments](https://mlsanity.com/paper/2309.03651/) (2023)
- [Searching for Programmatic Policies in Semantic Spaces](https://mlsanity.com/paper/2405.05431/) (2024)
- [Learning big logical rules by joining small rules](https://mlsanity.com/paper/2401.16215/) (2024)
- [3D Building Generation in Minecraft via Large Language Models](https://mlsanity.com/paper/2406.08751/) (2024)
- [KnowPC: Knowledge-Driven Programmatic Reinforcement Learning for Zero-shot Coordination](https://mlsanity.com/paper/2408.04336/) (2024)
- [A Genetic Programming Framework for 2D Platform AI](https://mlsanity.com/paper/1803.01648/) (2018)
- [Software Engineering For Automated Game Design](https://mlsanity.com/paper/2004.01770/) (2020)

## Built on by (most cited first, 5 of 5)

- [Evaluating LLMs in Open-Source Games](https://mlsanity.com/paper/2512.00371/) (2025)
- [Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning](https://mlsanity.com/paper/2607.22732/) (2026)
- [Observer, Not Player: Simulating Theory of Mind in LLMs through Game Observation](https://mlsanity.com/paper/2512.19210/) (2025)
- [Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver](https://mlsanity.com/paper/2604.25067/) (2026)
- [Can AI Agents Learn Their Way to the Top? Evaluating Heuristic Learning in a Long-Running Game Agent Competition](https://mlsanity.com/paper/2610.12341/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown impressive capabilities in generating program code, opening exciting opportunities for applying program synthesis to games. In this work, we explore the potential of LLMs to directly synthesize usable code for a wide range of gaming applications, focusing on two programming languages, Python and Java. We use an evolutionary hill-climbing algorithm, where the mutations and seeds of the initial programs are controlled by LLMs. For Python, the framework covers various game-related tasks, including five miniature versions of Atari games, ten levels of Baba is You, an environment inspired by Asteroids, and a maze generation task. For Java, the framework contains 12 games from the TAG tabletop games framework. Across 29 tasks, we evaluated 12 language models for Python and 8 for Java. Our findings suggest that the performance of LLMs depends more on the task than on model size. While larger models generate more executable programs, these do not always result in higher-quality solutions but are much more expensive. No model has a clear advantage, although on any specific task, one model may be better. Trying many models on a problem and using the best results across them is more reliable than using just one.