# SPRITE: From Static Mockups to Engine-Ready Game UI

arXiv: https://arxiv.org/abs/2604.18591 · HTML: https://mlsanity.com/paper/2604.18591/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Builds on (most cited first, 19 of 19)

- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [PubLayNet: largest dataset ever for document layout analysis](https://mlsanity.com/paper/1908.07836/) (2019)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](https://mlsanity.com/paper/2303.08137/) (2023)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [LayoutTransformer: Layout Generation and Completion with Self-attention](https://mlsanity.com/paper/2006.14615/) (2020)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Deep Learning for Source Code Modeling and Generation: Models, Applications and Challenges](https://mlsanity.com/paper/2002.05442/) (2020)
- [READ: Recursive Autoencoders for Document Layout Generation](https://mlsanity.com/paper/1909.00302/) (2019)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach](https://mlsanity.com/paper/2406.16386/) (2024)
- [LayoutGMN: Neural Graph Matching for Structural Layout Similarity](https://mlsanity.com/paper/2012.06547/) (2020)
- [MLLM-Based UI2Code Automation Guided by UI Layout Information](https://mlsanity.com/paper/2506.10376/) (2025)
- [Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation](https://mlsanity.com/paper/2508.20265/) (2025)
- [Misty: UI Prototyping Through Interactive Conceptual Blending](https://mlsanity.com/paper/2409.13900/) (2024)
- [Widget2Code: From Visual Widgets to UI Code via Multimodal LLMs](https://mlsanity.com/paper/2512.19918/) (2025)
- [Erratum: Leveraging Flexible Tree Matching to Repair Broken Locators in Web Automation Scripts](https://mlsanity.com/paper/2106.04916/) (2021)

## Abstract (arXiv)

Game UI implementation requires translating stylized mockups into interactive engine entities. However, current "Screenshot-to-Code" tools often struggle with the irregular geometries and deep visual hierarchies typical of game interfaces. To bridge this gap, we introduce SPRITE, a pipeline that transforms static screenshots into editable engine assets. By integrating Vision-Language Models (VLMs) with a structured YAML intermediate representation, SPRITE explicitly captures complex container relationships and non-rectangular layouts. We evaluated SPRITE against a curated Game UI benchmark and conducted expert reviews with professional developers to assess reconstruction fidelity and prototyping efficiency. Our findings demonstrate that SPRITE streamlines development by automating tedious coding and resolving complex nesting. By facilitating rapid in-engine iteration, SPRITE effectively blurs the boundaries between artistic design and technical implementation in game development. Project page: https://baiyunshu.github.io/sprite.github.io/