# Leveraging Multimodal LLM for Inspirational User Interface Search

arXiv: https://arxiv.org/abs/2501.17799 · HTML: https://mlsanity.com/paper/2501.17799/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2025.

## Built on by (most cited first, 13 of 13)

- [Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025](https://mlsanity.com/paper/2506.12430/) (2025)
- [DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfaces](https://mlsanity.com/paper/2509.13444/) (2025)
- [Personalize Your Large Vision-language Models With In-context Prompt Tuning](https://mlsanity.com/paper/2605.31513/) (2026)
- [CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement](https://mlsanity.com/paper/2512.24947/) (2025)
- [AlignUI: A Method for Designing LLM-Generated UIs Aligned with User Preferences](https://mlsanity.com/paper/2601.17614/) (2026)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Bridging Gulfs in UI Generation through Semantic Guidance](https://mlsanity.com/paper/2601.19171/) (2026)
- [GhostUI: Unveiling Hidden Interactions in Mobile UI](https://mlsanity.com/paper/2601.19258/) (2026)
- [PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactions](https://mlsanity.com/paper/2601.15575/) (2026)
- [Personagram: Bridging Personas and Product Design for Creative Ideation with Multimodal LLMs](https://mlsanity.com/paper/2602.06197/) (2026)
- [ReFinE: Streamlining UI Mockup Iteration with Research Findings](https://mlsanity.com/paper/2604.04353/) (2026)
- [Disentangling Visual and Factual Correctness in LVLMs' Visualization Literacy](https://mlsanity.com/paper/2606.03142/) (2026)
- [Motif: Discovering and Automating Personal Web Workflows](https://mlsanity.com/paper/2607.10531/) (2026)

## Abstract (arXiv)

Inspirational search, the process of exploring designs to inform and inspire new creative work, is pivotal in mobile user interface (UI) design. However, exploring the vast space of UI references remains a challenge. Existing AI-based UI search methods often miss crucial semantics like target users or the mood of apps. Additionally, these models typically require metadata like view hierarchies, limiting their practical use. We used a multimodal large language model (MLLM) to extract and interpret semantics from mobile UI images. We identified key UI semantics through a formative study and developed a semantic-based UI search system. Through computational and human evaluations, we demonstrate that our approach significantly outperforms existing UI retrieval methods, offering UI designers a more enriched and contextually relevant search experience. We enhance the understanding of mobile UI design semantics and highlight MLLMs' potential in inspirational search, providing a rich dataset of UI semantics for future studies.