# Investigating Multimodal Large Language Models to Support Usability Evaluation

arXiv: https://arxiv.org/abs/2508.16165 · HTML: https://mlsanity.com/paper/2508.16165/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 21 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Builds on (most cited first, 4 of 4)

- [Generating Automatic Feedback on UI Mockups with Large Language Models](https://mlsanity.com/paper/2403.13139/) (2024)
- [Does GenAI Make Usability Testing Obsolete?](https://mlsanity.com/paper/2411.00634/) (2024)
- [Can GPT-4o Evaluate Usability Like Human Experts? A Comparative Study on Issue Identification in Heuristic Evaluation](https://mlsanity.com/paper/2506.16345/) (2025)
- [A Comprehensive Model of Usability](https://mlsanity.com/paper/1612.04598/) (2016)

## Abstract (arXiv)

Usability evaluation is an essential method to support the design of effective and intuitive user interfaces (UIs). However, it commonly relies on resource-intensive, expert-driven methods, which limit its accessibility, especially for small organizations. Recent multimodal large language models (MLLMs) have the potential to support usability evaluation by analyzing textual instructions together with visual UI context. This paper investigates the use of MLLMs as assistive tools for usability evaluation by framing the task as a prioritization problem. It identifies and explains usability issues and ranks them by severity. We report a study that compares the evaluations generated by multiple MLLMs with assessments from usability experts. The results demonstrate that MLLMs can offer complementary insights and support the efficient prioritization of critical issues. Additionally, we present an interactive visualization tool that enables the transparent review and validation of model-generated findings. Based on this, we outline concepts for integrating MLLM-based usability evaluation into real-world development workflows.