# On AI-Inspired UI-Design

arXiv: https://arxiv.org/abs/2406.13631 · HTML: https://mlsanity.com/paper/2406.13631/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 18 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Boosting GUI Prototyping with Diffusion Models](https://mlsanity.com/paper/2306.06233/) (2023)
- [Design Patterns for Machine Learning Based Systems with Human-in-the-Loop](https://mlsanity.com/paper/2312.00582/) (2023)
- [Designing with Language: Wireframing UI Design Intent with Generative Large Language Models](https://mlsanity.com/paper/2312.07755/) (2023)
- [GUing: A Mobile GUI Search Engine using a Vision-Language Model](https://mlsanity.com/paper/2405.00145/) (2024)
- [Automatically Distilling Storyboard with Rich Features for Android Apps](https://mlsanity.com/paper/2203.06420/) (2022)

## Built on by (most cited first, 4 of 4)

- [ViMo: A Generative Visual GUI World Model for App Agents](https://mlsanity.com/paper/2504.13936/) (2025)
- [The role of large language models in UI/UX design: A systematic literature review](https://mlsanity.com/paper/2507.04469/) (2025)
- [Getting Inspiration for Feature Elicitation: App Store- vs. LLM-based Approach](https://mlsanity.com/paper/2408.17404/) (2024)
- [How Well Can AI Generate Backlogs from App Mockups?](https://mlsanity.com/paper/2607.22902/) (2026)

## Abstract (arXiv)

Graphical User Interface (or simply UI) is a primary mean of interaction between users and their devices. In this paper, we discuss three complementary Artificial Intelligence (AI) approaches for triggering the creativity of app designers and inspiring them create better and more diverse UI designs. First, designers can prompt a Large Language Model (LLM) to directly generate and adjust UIs. Second, a Vision-Language Model (VLM) enables designers to effectively search a large screenshot dataset, e.g. from apps published in app stores. Third, a Diffusion Model (DM) can be trained to specifically generate UIs as inspirational images. We present an AI-inspired design process and discuss the implications and limitations of the approaches.