# SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI

arXiv: https://arxiv.org/abs/2509.07334 · HTML: https://mlsanity.com/paper/2509.07334/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2025.

## Built on by (most cited first, 3 of 3)

- [GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation](https://mlsanity.com/paper/2603.14724/) (2026)
- [Portal UX Agent -- A Plug-and-Play Engine for Rendering UIs from Natural Language Specifications](https://mlsanity.com/paper/2511.00843/) (2025)
- [Efficient Personalization of Generative User Interfaces](https://mlsanity.com/paper/2604.09876/) (2026)

## Abstract (arXiv)

Large language models (LLMs) promise to accelerate UI design, yet current tools struggle with two fundamentals: externalizing designers' intent and controlling iterative change. We introduce SPEC, a structured, parameterized, hierarchical intermediate representation that exposes UI elements as controllable parameters. Building on SPEC, we present SpecifyUI, an interactive system that extracts SPEC from UI references via region segmentation and vision-language models, composes UIs across multiple sources, and supports targeted edits at global, regional, and component levels. A multi-agent generator renders SPEC into high-fidelity designs, closing the loop between intent expression and controllable generation. Quantitative experiments show SPEC-based generation more faithfully captures reference intent than prompt-based baselines. In a user study with 16 professional designers, SpecifyUI significantly outperformed Stitch on intent alignment, design quality, controllability, and overall experience in human-AI co-creation. Our results position SPEC as a specification-driven paradigm that shifts LLM-assisted design from one-shot prompting to iterative, collaborative workflows.