mlsanityData to 14 Oct 2026

Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding

arXiv 2505.05026 ↗First posted 8 May 2025cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

User interface (UI) design goes beyond visuals to shape user experience (UX), underscoring the shift toward UI/UX as a unified concept. While recent studies have explored UI evaluation using Multimodal Large Language Models (MLLMs), they largely focus on surface-level features, overlooking how design choices influence user behavior at scale. To fill this gap, we introduce WiserUI-Bench, a novel benchmark for multimodal understanding of how UI/UX design affects user behavior, built on 300 real-world UI image pairs from industry A/B tests, with empirically validated winners that induced more user actions. For future design progress in practice, post-hoc understanding of why such winners succeed with mass users is also required; we support this via expert-curated key interpretations for each instance. Experiments across multiple MLLMs on WiserUI-Bench for two main tasks, (1) predicting the more effective UI image between an A/B-tested pair, and (2) explaining it post-hoc in alignment with expert interpretations, show that models exhibit limited understanding of the behavioral impact of UI/UX design. We believe our work will foster research on leveraging MLLMs for visual design in user behavior contexts.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 25 of its 63 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

25 of 63 references.

References · 25

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
13 more
  1. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  2. 2019Abductive Commonsense Reasoningcited by 301
  3. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 134
  4. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 129
  5. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  6. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 72
  7. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  8. 2024CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputscited by 56
  9. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
  10. 2024UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedbackcited by 27
  11. 2024UICrit: Enhancing Automated Design Evaluation with a UICritique Datasetcited by 21
  12. 2025AgentA/B: Automated and Scalable Web A/BTesting with Interactive LLM Agentscited by 19
  13. 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces
  2. 2026Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
  3. 2025CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design
  4. 2025Leveraging Multimodal LLM for Inspirational User Interface Search
  5. 2025Investigating Multimodal Large Language Models to Support Usability Evaluation
  6. 2026Mapping the Design Space of User Experience for Computer Use Agents
  7. 2026Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations
  8. 2025UI-UG: A Unified MLLM for UI Understanding and Generation