mlsanityData to 14 Oct 2026

JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines

arXiv 2606.19830 ↗First posted 18 Jun 2026cs.SE cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on the Godot engine's text-based format and headless execution mode, we design a deterministic verification pipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combining compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

27 of 38 references.

References · 27

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  5. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  6. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  7. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  8. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  9. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 152
  10. 2017Procedural Content Generation via Machine Learning (PCGML)cited by 137
  11. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 117
  12. 2024A Survey on Large Language Model-Based Game Agentscited by 112
15 more
  1. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  2. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  3. 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 64
  4. 2025Automated Benchmark Generation for Repository-Level Coding Taskscited by 29
  5. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generationcited by 26
  6. 2024Procedural Content Generation in Games: A Survey with Insights on Emerging LLM Integrationcited by 22
  7. 2025DSCodeBench: A Realistic Benchmark for Data Science Code Generationcited by 17
  8. 2024DreamGarden: A Designer Assistant for Growing Games from a Single Promptcited by 10
  9. 2024Tiled Diffusioncited by 9
  10. 2025GameTileNet: A Semantic Dataset for Low-Resolution Game Art in Procedural Content Generationcited by 3
  11. 2026ProxyWar: Dynamic Assessment of LLM Code Generation in Game Arenascited by 3
  12. 2026RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practicescited by 3
  13. 2014Promoting Inclusive Design Practice at the Global Game Jam: A Pilot Evaluationcited by 1
  14. 2021Two Decades of Game Jamscited by 1
  15. 2024Sprite Sheet Diffusion: Generate Game Character for Animationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026The Verifier is the Curriculum: Precision Sets the Return on Search in Code Self-Distillationcited by 3
  2. 2026AI for Games in the Foundation Model Eracited by 3
  3. 2026GameASG-Bench: Benchmarking Autonomous Software Generation for Game Developmentcited by 1
  4. 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agentscited by 0
  5. 2026SWE-Game: Can Coding Agents Build the Games We Want?cited by 0
  6. 2026GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assetscited by 0