# GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development

arXiv: https://arxiv.org/abs/2609.21293 · HTML: https://mlsanity.com/paper/2609.21293/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 44 references that are in the map.
- First posted on arXiv 18 Sep 2026.

## Builds on (most cited first, 30 of 39)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch](https://mlsanity.com/paper/2505.03733/) (2025)
- [FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation](https://mlsanity.com/paper/2503.06680/) (2025)
- [GameGPT: Multi-agent Collaborative Framework for Game Development](https://mlsanity.com/paper/2310.08067/) (2023)
- [GameDevBench: Evaluating Agentic Capabilities Through Game Development](https://mlsanity.com/paper/2602.11103/) (2026)
- [Game Generation via Large Language Models](https://mlsanity.com/paper/2404.08706/) (2024)
- [GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?](https://mlsanity.com/paper/2606.17861/) (2026)
- [OpenGame: Open Agentic Coding for Games](https://mlsanity.com/paper/2604.18394/) (2026)
- [GUI Agents for Continual Game Generation](https://mlsanity.com/paper/2605.28258/) (2026)
- [CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation](https://mlsanity.com/paper/2504.21751/) (2025)
- [Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification](https://mlsanity.com/paper/2603.26648/) (2026)
- [WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games](https://mlsanity.com/paper/2605.17637/) (2026)
- [PlayCoder: Making LLM-Generated GUI Code Playable](https://mlsanity.com/paper/2604.19742/) (2026)
- [GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection](https://mlsanity.com/paper/2605.07442/) (2026)
- [V-GameGym: Visual Game Generation for Code Large Language Models](https://mlsanity.com/paper/2509.20136/) (2025)
- [AutoUE: Automated Generation of 3D Games in Unreal Engine via Multi-Agent Systems](https://mlsanity.com/paper/2603.07106/) (2026)
- [GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments](https://mlsanity.com/paper/2607.03525/) (2026)
- [GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?](https://mlsanity.com/paper/2608.21833/) (2026)
- [E2Edev: Benchmarking Large Language Models in End-to-End Software Development Task](https://mlsanity.com/paper/2510.14509/) (2025)
- [JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines](https://mlsanity.com/paper/2606.19830/) (2026)
- [KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?](https://mlsanity.com/paper/2601.13240/) (2026)
- [Distilling Game Code World Model Generation into Lightweight Large Language Models](https://mlsanity.com/paper/2605.24375/) (2026)

## Built on by (most cited first, 1 of 1)

- [Recursive Game Creator: An Agentic Product-Level Experience-Oriented Game Harness](https://mlsanity.com/paper/2610.08621/) (2026)

## Abstract (arXiv)

Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessarily establish that their interacting components satisfy the specified behavioral requirements. We introduce GameASG-Bench, a benchmark that makes behavioral testability part of the generation task for game development. Our design declares an evaluation interface specification before generation, fixing legal starting scenarios, player-level actions, stable snapshots, rejection behavior, and invariants while leaving private implementations open. Concretely, we include: (i) static L1 checks that assess source-level compliance; and (ii) browser-executed L2 checks that combine semantic observations with real input and runtime evidence. We implement this protocol as 47 browser-native game-generation tasks spanning 12 primary genres and both 2D and 3D interaction, each with executable checks and an independently verified reference implementation. Our experiments answer four key questions about end-to-end agent performance, tool access and nominal turn budget, reasoning effort, and harness choice. Across nine agent stacks, the highest observed mean L2 check pass rate is 93.2%, yet the highest observed strict task success rate, requiring all L1 and applicable L2 prerequisite and core requirement checks, is only 55.3% (26/47 tasks). For DeepSeek-V4-Flash, full tool access and larger nominal turn budgets yield more strict task successes, while the strict task success rate is not monotonic in reasoning effort. Both tested harnesses achieve 18 strict task successes, but only ten tasks succeed under both. These results expose task-level compliance gaps that high average check pass rates actually obscure.