mlsanityData to 7 Oct 2026

PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation

arXiv 2606.05697 ↗First posted 4 Jun 2026cs.AIcited by 1

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

User interface (UI) and user experience (UX) evaluation is central to product development, yet reliable feedback still relies on recruiting human participants or running online A/B tests, making early-stage iteration slow and costly. In light of this, recent work has explored Multimodal Large Language Models as proxy evaluators. However, existing approaches either produce surface-level critiques or a judgment that reflects the model's own biases rather than the genuine response of a particular user. We introduce PerceptUI, a framework for persona-conditioned UI/UX evaluation that predicts how a specific user would answer interface-related questions and produces natural-language rationales. PerceptUI is trained in two stages: (i) contrastive reflection fine-tuning distills teacher-generated rationales by extracting lessons from human decisions, and (ii) a reflective prompt-evolution step from the model's own failure traces. Across multiple domains and datasets, PerceptUI achieves human-level realism, generalizes to unseen questions and personas, and yields population-level response distributions.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 35 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

22 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Mapping the Design Space of User Experience for Computer Use Agents7 Feb
  2. 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents13 Apr
  3. 2025UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design18 Feb

Other references · 19

  1. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  3. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 228
  4. 2025CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulationcited by 28
  5. 2024UICrit: Enhancing Automated Design Evaluation with a UICritique Datasetcited by 21
  6. 2025MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfacescited by 11
  7. 2026AlignUSER: Human-Aligned LLM Agents via World Models for Recommender System Evaluationcited by 6
  8. 2024Visual Prompting with Iterative Refinement for Design Critique Generationcited by 5
  9. 2025AI Agents for Web Testing: A Case Study in the Wildcited by 5
  10. 2025Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluationcited by 4
  11. 2026UXCascade: Scalable Usability Testing with Simulated User Agentscited by 4
  12. 2025Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understandingcited by 3
7 more
  1. 2025Improving User Interface Generation Models from Designer Feedbackcited by 3
  2. 2026SimAB: Simulating A/B Tests with Persona-Conditioned AI Agents for Rapid Design Evaluationcited by 3
  3. 2026Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluationcited by 3
  4. 2025Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluationcited by 2
  5. 2026CritiqueCrew: Orchestrating Multi-Perspective Conversational Design Critiquecited by 2
  6. 2026Avenir-UX: Automated UX Evaluation via Simulated Human Web Interaction with GUI Groundingcited by 2
  7. 2025Synthetic Cognitive Walkthrough: Aligning Large Language Model Performance with Human Cognitive Walkthroughcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026UXBench: Benchmarking User Experience in AI Assistants
  2. 2026Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
  3. 2026Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
  4. 2026Automatic multimodal UX improvement recommendations from LLM agent user simulations
  5. 2026Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User Interfaces
  6. 2026UXSim: Towards a Hybrid User Search Simulation
  7. 2026MatrAIx: Simulating the World with 8.3 Billion Persona Agents
  8. 2025UI-UG: A Unified MLLM for UI Understanding and Generation