mlsanityData to 7 Oct 2026

Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

arXiv 2606.13192 ↗First posted 11 Jun 2026cs.AIcited by 1

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI). The application of multimodal large language models (MLLMs) in the field of user interfaces is evolving rapidly, such as visual element grounding, graphical user interface (GUI) agents, and design-to-code generation. However, research efforts on evaluating UX based on UI screenshots are still immature. To address this, we propose UXBench, a novel multimodal benchmark consisting of 2,000 VQA data samples designed to assess MLLMs' ability to perform UI-based reasoning. UXBench includes 8 tasks based on real-world UI screenshots that require fine-grained diagnosis of UX issues across layout relationships, visual hierarchy, and content consistency. Our extensive evaluation of mainstream MLLMs shows that they remain fundamentally limited in their capacity for UI-based reasoning. The results underscore the need for further advancements in this area. To bridge this gap, we propose UI-UX, an MLLM based on Qwen3-VL-4B-Thinking foundation model and enhanced via reinforcement learning with two key innovations: a reward routing mechanism that dynamically balances perceptual understanding and logical reasoning during inference, and an asymmetric transition reward that suppresses redundant or insufficient reasoning steps. Experiments demonstrate that UI-UX achieves state-of-the-art (SOTA) performance on UXBench, attaining an accuracy of 0.7963 -- surpassing Claude-4.5-Sonnet's 0.6550 -- while exhibiting strong generalization across diverse UI tasks and maintaining low inference latency.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 33 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

27 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency25 Aug

Other references · 26

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  11. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  12. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
14 more
  1. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
  2. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  3. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agentscited by 56
  4. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Modelscited by 43
  5. 2024Harnessing Webpage UIs for Text-Rich Visual Understandingcited by 28
  6. 2023UI Layout Generation with LLMs Guided by UI Grammarcited by 15
  7. 2022Nighthawk: Fully Automated Localizing UI Display Issues via Visual Understandingcited by 6
  8. 2025Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Modelscited by 4
  9. 2024Large Language Models for Mobile GUI Text Input Generation: An Empirical Studycited by 3
  10. 2025CoreEval: Automatically Building Contamination-Resilient Datasets with Real-World Knowledge toward Reliable LLM Evaluationcited by 3
  11. 2022The Metamorphosis: Automatic Detection of Scaling Issues for Mobile Appscited by 2
  12. 2025The role of large language models in UI/UX design: A systematic literature reviewcited by 2
  13. 2018Beyond "Usability and User Experience" , Towards an Integrative Heuristic Inspection: from Accessibility to Persuasiveness in the UX Evaluation A Case Study on an Insurance Prospecting Tablet Applicationcited by 1
  14. 2021OwlEyes-Online: A Fully Automated Platform for Detecting and Localizing UI Display Issuescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding
  2. 2025MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces
  3. 2025CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design
  4. 2025Investigating Multimodal Large Language Models to Support Usability Evaluation
  5. 2025UI-UG: A Unified MLLM for UI Understanding and Generation
  6. 2025Leveraging Multimodal LLM for Inspirational User Interface Search
  7. 2026UXBench: Benchmarking User Experience in AI Assistants
  8. 2026PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation