GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation
Not on the map. Why?
Abstract (arXiv)
Game UI design requires consistent visual assets across rarity tiers yet remains a predominantly manual process. We present GameUIAgent, an LLM-powered agentic framework that translates natural language descriptions into editable Figma designs via a Design Spec JSON intermediate representation. A six-stage neuro-symbolic pipeline combines LLM generation, deterministic post-processing, and a Vision-Language Model (VLM)-guided Reflection Controller (RC) for iterative self-correction with guaranteed non-regressive quality. Evaluated across 110 test cases, three LLMs, and three UI templates, cross-model analysis establishes a game-domain failure taxonomy (rarity-dependent degradation; visual emptiness) and uncovers two key empirical findings. A Quality Ceiling Effect (Pearson r=-0.96, p<0.01) suggests that RC improvement is bounded by headroom below a quality threshold -- a visual-domain counterpart to test-time compute scaling laws. A Rendering-Evaluation Fidelity Principle reveals that partial rendering enhancements paradoxically degrade VLM evaluation by amplifying structural defects. Together, these results establish foundational principles for LLM-driven visual generation agents in game production.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 16 Mar 2026.
Builds on
17 of 31 references.
References · 17
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
- 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
- 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
- 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 113
- 2024Large Language Models and Games: A Survey and Roadmapcited by 91
5 more
- 2026Large Language Model Agents Are Not Always Faithful Self-Evolverscited by 5
- 2025GUIDE: LLM-Driven GUI Generation Decomposition for Automated Prototypingcited by 4
- 2025SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AIcited by 3
- 2026CoGen: Creation of Reusable UI Components in Figma via Textual Commandscited by 2
- 2024AutoGameUI: Constructing High-Fidelity GameUI via Multimodal Correspondence Matchingcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Code2Games: Enabling Coding Agents for Gaming World Generation
- 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models
- 2026GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents
- 2026OpenGame: Open Agentic Coding for Games
- 2026GameDevBench: Evaluating Agentic Capabilities Through Game Development
- 2025Automated Unity Game Template Generation from GDDs via NLP and Multi-Modal LLMs
- 2026GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
- 2025Portal UX Agent -- A Plug-and-Play Engine for Rendering UIs from Natural Language Specifications