Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding
Not on the map. Why?
Abstract (arXiv)
User interface (UI) design goes beyond visuals to shape user experience (UX), underscoring the shift toward UI/UX as a unified concept. While recent studies have explored UI evaluation using Multimodal Large Language Models (MLLMs), they largely focus on surface-level features, overlooking how design choices influence user behavior at scale. To fill this gap, we introduce WiserUI-Bench, a novel benchmark for multimodal understanding of how UI/UX design affects user behavior, built on 300 real-world UI image pairs from industry A/B tests, with empirically validated winners that induced more user actions. For future design progress in practice, post-hoc understanding of why such winners succeed with mass users is also required; we support this via expert-curated key interpretations for each instance. Experiments across multiple MLLMs on WiserUI-Bench for two main tasks, (1) predicting the more effective UI image between an A/B-tested pair, and (2) explaining it post-hoc in alignment with expert interpretations, show that models exhibit limited understanding of the behavioral impact of UI/UX design. We believe our work will foster research on leveraging MLLMs for visual design in user behavior contexts.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 25 of its 63 references that are in the map. First posted on arXiv 8 May 2025.
Builds on
25 of 63 references.
References · 25
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
13 more
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
- 2019Abductive Commonsense Reasoningcited by 301
- 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
- 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 129
- 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
- 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 72
- 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
- 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
- 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
- 2024UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedbackcited by 27
- 2024UICrit: Enhancing Automated Design Evaluation with a UICritique Datasetcited by 21
- 2025AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agentscited by 19
- 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces
- 2026Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
- 2025CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design
- 2025Leveraging Multimodal LLM for Inspirational User Interface Search
- 2025Investigating Multimodal Large Language Models to Support Usability Evaluation
- 2026Mapping the Design Space of User Experience for Computer Use Agents
- 2026Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations
- 2025UI-UG: A Unified MLLM for UI Understanding and Generation