mlsanityData to 7 Oct 2026

Mage: Multi-Axis Evaluation of LLM-Generated Executable Game Scenes Beyond Compile-Pass Rate

arXiv 2605.07342 ↗First posted 8 May 2026cs.LG cs.AI cs.SEcited by 4

Not on the map. Why?

Abstract (arXiv)

Compile-pass rate is the dominant evaluation signal for LLM code generation, yet for multi-component domain-specific artifacts it can be actively misleading. We demonstrate this on executable game scene synthesis with a four-axis evaluation protocol (named `Mage') -- compile success, runtime success, structural fidelity, and mechanism adherence -- applied to 858 generation attempts across four open-weight LLMs (7B--30B), 26~hand-crafted Unity goal pattern playable concepts, and two automatically extracted IR granularity levels. Direct NL-to-C# generation achieves the highest runtime-pass rate (43% mean) yet produces structurally vacuous scenes (mechanism F1 ≈ 0.12). Structural IR conditioning halves the runtime rate but recovers domain-faithful structure (F1 up to 1.00). Within IR conditioning, behavior-only and full-scene granularity are statistically indistinguishable (McNemar p = 1.0), indicating input-level granularity saturation. These results show that compile rate is anti-correlated with functional correctness in this domain and that multi-axis evaluation is necessary to detect the divergence. We release the benchmark, replay logs, and per-record metrics for independent verification.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

20 of 28 references.

References · 20

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  9. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  10. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  11. 2023Augmented Language Models: a Surveycited by 398
  12. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
8 more
  1. 2023Efficient Guided Generation for Large Language Modelscited by 192
  2. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
  3. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  4. 2024SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Modelcited by 95
  5. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  6. 2023Level Generation Through Large Language Modelscited by 68
  7. 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 63
  8. 2026Grounding Machine Creativity in Game Design Knowledge Representations: Empirical Probing of LLM-Based Executable Synthesis of Goal Playable Patterns under Structural Constraintscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.