# A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

arXiv: https://arxiv.org/abs/2609.39564 · HTML: https://mlsanity.com/paper/2609.39564/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 41 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 22 of 22)

- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026)
- [ProgramBench: Can Language Models Rebuild Programs From Scratch?](https://mlsanity.com/paper/2605.03546/) (2026)
- [GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents](https://mlsanity.com/paper/2604.07429/) (2026)
- [GameDevBench: Evaluating Agentic Capabilities Through Game Development](https://mlsanity.com/paper/2602.11103/) (2026)
- [GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?](https://mlsanity.com/paper/2606.17861/) (2026)
- [OpenGame: Open Agentic Coding for Games](https://mlsanity.com/paper/2604.18394/) (2026)
- [GUI Agents for Continual Game Generation](https://mlsanity.com/paper/2605.28258/) (2026)
- [WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games](https://mlsanity.com/paper/2605.17637/) (2026)
- [Automatic Detection of Causality in Requirement Artifacts: the CiRA Approach](https://mlsanity.com/paper/2101.10766/) (2021)
- [GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection](https://mlsanity.com/paper/2605.07442/) (2026)
- [GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments](https://mlsanity.com/paper/2607.03525/) (2026)
- [WebVR: Benchmarking Multimodal LLMs for WebPage Recreation from Videos via Human-Aligned Visual Rubrics](https://mlsanity.com/paper/2603.13391/) (2026)
- [MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation](https://mlsanity.com/paper/2605.28579/) (2026)
- [GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?](https://mlsanity.com/paper/2608.21833/) (2026)
- [OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics](https://mlsanity.com/paper/2606.09826/) (2026)
- [WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts](https://mlsanity.com/paper/2606.03220/) (2026)
- [LiveEvalBench: Toward Open-World Evaluation for Web Generation](https://mlsanity.com/paper/2608.03689/) (2026)
- [WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader](https://mlsanity.com/paper/2608.06474/) (2026)

## Abstract (arXiv)

Delegating complete application development to coding agents requires preserving the intended design rather than simply producing plausible outputs through naive prompting. Game development provides a demanding testbed, as long-form Game Design Documents (GDDs) describe requirements that must work together across game logic, visual rendering, and player interactions. However, existing game-development benchmarks typically use compact specifications and provide limited support for evaluating interdependent requirements across these aspects in long-form GDDs. We introduce A2Z GameSpec-Bench, a benchmark of 100 long-form GDDs for evaluating end-to-end game development by agents. We measure faithfulness by checking whether the game satisfies the GDD requirements and preserves the relationships among them. Each GDD is turned into a dependency-aware contract that contains rules, constraints, and prerequisite relations. Following game-development practices, we combine source-code inspection with agent-generated test policies for scenario-based replay and adaptive playtesting. The contract remains fixed across agents and revision rounds, while judgments and evidence linked to the same requirements support consistent comparison and failure detection. Our evaluations show that current agents struggle to jointly satisfy interdependent requirements across code implementation and actual play. Requirement-specific feedback improves GDD Fidelity by 10.9% relative to self-revision after two rounds. A2Z GameSpec-Bench assesses end-to-end specification-following ability beyond implementation judgments and provides targeted feedback to support more faithful game development. Code and datasets are available at https://a2z-gamespec-bench.github.io.