Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate
Not on the map. Why?
Abstract (arXiv)
Compile-pass rate is the dominant evaluation signal for LLM code generation, yet for multi-component domain-specific artifacts it can be actively misleading. We demonstrate this on executable game scene synthesis with a four-axis evaluation protocol (named `Mage') -- compile success, runtime success, structural fidelity, and mechanism adherence -- applied to 858 generation attempts across four open-weight LLMs (7B--30B), 26~hand-crafted Unity goal pattern playable concepts, and two automatically extracted IR granularity levels. Direct NL-to-C# generation achieves the highest runtime-pass rate (43% mean) yet produces structurally vacuous scenes (mechanism F1 ≈ 0.12). Structural IR conditioning halves the runtime rate but recovers domain-faithful structure (F1 up to 1.00). Within IR conditioning, behavior-only and full-scene granularity are statistically indistinguishable (McNemar p = 1.0), indicating input-level granularity saturation. These results show that compile rate is anti-correlated with functional correctness in this domain and that multi-axis evaluation is necessary to detect the divergence. We release the benchmark, replay logs, and per-record metrics for independent verification.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
20 of 28 references.
References · 20
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2024Qwen2 Technical Reportcited by 2,076
- 2024Qwen2.5-Coder Technical Reportcited by 1,151
- 2021Measuring Coding Challenge Competence With APPScited by 879
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
- 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
- 2023Augmented Language Models: a Surveycited by 398
- 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
8 more
- 2023Efficient Guided Generation for Large Language Modelscited by 192
- 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
- 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
- 2024SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Modelcited by 95
- 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
- 2023Level Generation Through Large Language Modelscited by 68
- 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 63
- 2026Grounding Machine Creativity in Game Design Knowledge Representations: Empirical Probing of LLM-Based Executable Synthesis of Goal Playable Patterns under Structural Constraintscited by 1
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026AI for Games in the Foundation Model Eracited by 2
- 2026Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes: A Compiler Error Census across 26 Goal Playable Conceptscited by 0
- 2026GameASG-Bench: Benchmarking Autonomous Software Generation for Game Developmentcited by 0
- 2026SWE-Game: Can Coding Agents Build the Games We Want?cited by 0
- 2026SWE-Game: Can Coding Agents Build the Games We Want?27 Sep 2026
- 2026GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development18 Sep 2026
- 2026AI for Games in the Foundation Model Era15 Sep 2026
- 2026Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes: A Compiler Error Census across 26 Goal Playable Concepts11 Jul 2026