# PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation

arXiv: https://arxiv.org/abs/2606.05697 · HTML: https://mlsanity.com/paper/2606.05697/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 22 of 22)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [S3: Social-network Simulation System with Large Language Model-Empowered Agents](https://mlsanity.com/paper/2307.14984/) (2023)
- [UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design](https://mlsanity.com/paper/2502.12561/) (2025)
- [CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation](https://mlsanity.com/paper/2506.21805/) (2025)
- [UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset](https://mlsanity.com/paper/2407.08850/) (2024)
- [MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces](https://mlsanity.com/paper/2510.08783/) (2025)
- [UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents](https://mlsanity.com/paper/2504.09407/) (2025)
- [Mapping the Design Space of User Experience for Computer Use Agents](https://mlsanity.com/paper/2602.07283/) (2026)
- [AI Agents for Web Testing: A Case Study in the Wild](https://mlsanity.com/paper/2509.05197/) (2025)
- [AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluation](https://mlsanity.com/paper/2601.00930/) (2026)
- [Visual Prompting with Iterative Refinement for Design Critique Generation](https://mlsanity.com/paper/2412.16829/) (2024)
- [Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation](https://mlsanity.com/paper/2506.16345/) (2025)
- [UXCascade: Scalable Usability Testing with Simulated User Agents](https://mlsanity.com/paper/2601.15777/) (2026)
- [Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding](https://mlsanity.com/paper/2505.05026/) (2025)
- [Improving User Interface Generation Models from Designer Feedback](https://mlsanity.com/paper/2509.16779/) (2025)
- [SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluation](https://mlsanity.com/paper/2603.01024/) (2026)
- [Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation](https://mlsanity.com/paper/2604.09549/) (2026)
- [Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation](https://mlsanity.com/paper/2507.02306/) (2025)
- [CritiqueCrew: Orchestrating Multi-Perspective Conversational Design Critique](https://mlsanity.com/paper/2602.01796/) (2026)
- [Avenir-UX: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding](https://mlsanity.com/paper/2604.09581/) (2026)
- [Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthrough](https://mlsanity.com/paper/2512.03568/) (2025)

## Built on by (most cited first, 1 of 1)

- [CityReal: Human-Aligned Urban Behavior and City Dynamics Simulation with Large-Scale LLM Agents](https://mlsanity.com/paper/2608.16897/) (2026)

## Abstract (arXiv)

User interface (UI) and user experience (UX) evaluation is central to product development, yet reliable feedback still relies on recruiting human participants or running online A/B tests, making early-stage iteration slow and costly. In light of this, recent work has explored Multimodal Large Language Models as proxy evaluators. However, existing approaches either produce surface-level critiques or a judgment that reflects the model's own biases rather than the genuine response of a particular user. We introduce PerceptUI, a framework for persona-conditioned UI/UX evaluation that predicts how a specific user would answer interface-related questions and produces natural-language rationales. PerceptUI is trained in two stages: (i) contrastive reflection fine-tuning distills teacher-generated rationales by extracting lessons from human decisions, and (ii) a reflective prompt-evolution step from the model's own failure traces. Across multiple domains and datasets, PerceptUI achieves human-level realism, generalizes to unseen questions and personas, and yields population-level response distributions.