# Generating Automatic Feedback on UI Mockups with Large Language Models

arXiv: https://arxiv.org/abs/2403.13139 · HTML: https://mlsanity.com/paper/2403.13139/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2024.

## Built on by (most cited first, 23 of 23)

- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset](https://mlsanity.com/paper/2407.08850/) (2024)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [VASCAR: Content-Aware Layout Generation via Visual-Aware Self-Correction](https://mlsanity.com/paper/2412.04237/) (2024)
- [Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulation](https://mlsanity.com/paper/2502.02780/) (2025)
- [DesignPref: Capturing Personal Preferences in Visual Design Generation](https://mlsanity.com/paper/2511.20513/) (2025)
- [Language Models as Critical Thinking Tools: A Case Study of Philosophers](https://mlsanity.com/paper/2404.04516/) (2024)
- [Visual Prompting with Iterative Refinement for Design Critique Generation](https://mlsanity.com/paper/2412.16829/) (2024)
- [Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaboration](https://mlsanity.com/paper/2509.10652/) (2025)
- [ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inference](https://mlsanity.com/paper/2509.14537/) (2025)
- [ChatGPT in Data Visualization Education: A Student Perspective](https://mlsanity.com/paper/2405.00748/) (2024)
- [Chart-to-Experience: Benchmarking Multimodal LLMs for Predicting Experiential Impact of Charts](https://mlsanity.com/paper/2505.17374/) (2025)
- [Improving User Interface Generation Models from Designer Feedback](https://mlsanity.com/paper/2509.16779/) (2025)
- [Accessibility Scout: Personalized Accessibility Scans of Built Environments](https://mlsanity.com/paper/2507.23190/) (2025)
- [StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation](https://mlsanity.com/paper/2508.03182/) (2025)
- [Computational Scaffolding of Composition, Value, and Color for Disciplined Drawing](https://mlsanity.com/paper/2509.17268/) (2025)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [DesignLab: Designing Slides Through Iterative Detection and Correction](https://mlsanity.com/paper/2507.17202/) (2025)
- [From Answer Givers to Design Mentors: Guiding LLMs with the Cognitive Apprenticeship Model](https://mlsanity.com/paper/2601.19053/) (2026)
- [Bridging Gulfs in UI Generation through Semantic Guidance](https://mlsanity.com/paper/2601.19171/) (2026)
- [Investigating Multimodal Large Language Models to Support Usability Evaluation](https://mlsanity.com/paper/2508.16165/) (2025)
- [ReFinE: Streamlining UI Mockup Iteration with Research Findings](https://mlsanity.com/paper/2604.04353/) (2026)
- [Efficient Personalization of Generative User Interfaces](https://mlsanity.com/paper/2604.09876/) (2026)

## Abstract (arXiv)

Feedback on user interface (UI) mockups is crucial in design. However, human feedback is not always readily available. We explore the potential of using large language models for automatic feedback. Specifically, we focus on applying GPT-4 to automate heuristic evaluation, which currently entails a human expert assessing a UI's compliance with a set of design guidelines. We implemented a Figma plugin that takes in a UI design and a set of written heuristics, and renders automatically-generated feedback as constructive suggestions. We assessed performance on 51 UIs using three sets of guidelines, compared GPT-4-generated design suggestions with those from human experts, and conducted a study with 12 expert designers to understand fit with existing practice. We found that GPT-4-based feedback is useful for catching subtle errors, improving text, and considering UI semantics, but feedback also decreased in utility over iterations. Participants described several uses for this plugin despite its imperfect suggestions.