# UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset

arXiv: https://arxiv.org/abs/2407.08850 · HTML: https://mlsanity.com/paper/2407.08850/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](https://mlsanity.com/paper/2303.08137/) (2023)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics](https://mlsanity.com/paper/2301.13280/) (2023)
- [Predicting Visual Importance Across Graphic Design Types](https://mlsanity.com/paper/2008.02912/) (2020)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [PLay: Parametrically Conditioned Layout Generation using Latent Diffusion](https://mlsanity.com/paper/2301.11529/) (2023)
- [Generating Automatic Feedback on UI Mockups with Large Language Models](https://mlsanity.com/paper/2403.13139/) (2024)
- [Reasons to Reject? Aligning Language Models with Judgments](https://mlsanity.com/paper/2312.14591/) (2023)
- [PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs](https://mlsanity.com/paper/2402.08657/) (2024)
- [Understanding Visual Saliency in Mobile User Interfaces](https://mlsanity.com/paper/2101.09176/) (2021)
- [GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback](https://mlsanity.com/paper/2001.05684/) (2020)
- [Optimizing User Interface Layouts via Gradient Descent](https://mlsanity.com/paper/2002.10702/) (2020)
- [Predicting Visual Attention in Graphic Design Documents](https://mlsanity.com/paper/2407.02439/) (2024)

## Built on by (most cited first, 21 of 21)

- [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025)
- [MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces](https://mlsanity.com/paper/2510.08783/) (2025)
- [DesignPref: Capturing Personal Preferences in Visual Design Generation](https://mlsanity.com/paper/2511.20513/) (2025)
- [Generative Interfaces for Language Models](https://mlsanity.com/paper/2508.19227/) (2025)
- [Visual Prompting with Iterative Refinement for Design Critique Generation](https://mlsanity.com/paper/2412.16829/) (2024)
- [PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels](https://mlsanity.com/paper/2504.16419/) (2025)
- [Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding](https://mlsanity.com/paper/2505.05026/) (2025)
- [Improving User Interface Generation Models from Designer Feedback](https://mlsanity.com/paper/2509.16779/) (2025)
- [ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models](https://mlsanity.com/paper/2504.02110/) (2025)
- [FeedQUAC: Quick Unobtrusive AI-Generated Commentary](https://mlsanity.com/paper/2504.16416/) (2025)
- [The role of large language models in UI/UX design: A systematic literature review](https://mlsanity.com/paper/2507.04469/) (2025)
- [Training Computer Use Agents to Assess the Usability of Graphical User Interfaces](https://mlsanity.com/paper/2604.26020/) (2026)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation](https://mlsanity.com/paper/2606.05697/) (2026)
- [LikeThis! Empowering App Users to Submit UI Improvement Suggestions Instead of Complaints](https://mlsanity.com/paper/2603.04245/) (2026)
- [Efficient Personalization of Generative User Interfaces](https://mlsanity.com/paper/2604.09876/) (2026)
- [Through the PRISM: Principle-Aware, Interpretable, and Multi-Scale Evaluation of Visual Designs](https://mlsanity.com/paper/2606.00592/) (2026)
- [SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation](https://mlsanity.com/paper/2608.22390/) (2026)
- [Automatic multimodal UX improvement recommendations from LLM agent user simulations](https://mlsanity.com/paper/2609.22971/) (2026)
- [A Benchmarking Framework for Context-aware XR Interfaces](https://mlsanity.com/paper/2609.30466/) (2026)
- [AUV-Bench: Aesthetic Understanding and Generation Evaluation for User Interfaces](https://mlsanity.com/paper/2609.34854/) (2026)

## Abstract (arXiv)

Automated UI evaluation can be beneficial for the design process; for example, to compare different UI designs, or conduct automated heuristic evaluation. LLM-based UI evaluation, in particular, holds the promise of generalizability to a wide variety of UI types and evaluation tasks. However, current LLM-based techniques do not yet match the performance of human evaluators. We hypothesize that automatic evaluation can be improved by collecting a targeted UI feedback dataset and then using this dataset to enhance the performance of general-purpose LLMs. We present a targeted dataset of 3,059 design critiques and quality ratings for 983 mobile UIs, collected from seven experienced designers. We carried out an in-depth analysis to characterize the dataset's features. We then applied this dataset to achieve a 55% performance gain in LLM-generated UI feedback via various few-shot and visual prompting techniques. We also discuss future applications of this dataset, including training a reward model for generative UI techniques, and fine-tuning a tool-agnostic multi-modal LLM that automates UI evaluation.