# Does GenAI Make Usability Testing Obsolete?

arXiv: https://arxiv.org/abs/2411.00634 · HTML: https://mlsanity.com/paper/2411.00634/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2024.

## Built on by (most cited first, 6 of 6)

- [Investigating Multimodal Large Language Models to Support Usability Evaluation](https://mlsanity.com/paper/2508.16165/) (2025)
- [The Software Engineering Simulations Lab: Agentic AI for RE Quality Simulations](https://mlsanity.com/paper/2511.17762/) (2025)
- [Catching UX Flaws in Code: Leveraging LLMs to Identify Usability Flaws at the Development Stage](https://mlsanity.com/paper/2512.04262/) (2025)
- [FeedAIde: Guiding App Users to Submit Rich Feedback Reports by Asking Context-Aware Follow-Up Questions](https://mlsanity.com/paper/2603.04244/) (2026)
- [LikeThis! Empowering App Users to Submit UI Improvement Suggestions Instead of Complaints](https://mlsanity.com/paper/2603.04245/) (2026)
- [How Well Can AI Generate Backlogs from App Mockups?](https://mlsanity.com/paper/2607.22902/) (2026)

## Abstract (arXiv)

Ensuring usability is crucial for the success of mobile apps. Usability issues can compromise user experience and negatively impact the perceived app quality. This paper presents UX-LLM, a novel tool powered by a Large Vision-Language Model that predicts usability issues in iOS apps. To evaluate the performance of UX-LLM, we predicted usability issues in two open-source apps of a medium complexity and asked two usability experts to assess the predictions. We also performed traditional usability testing and expert review for both apps and compared the results to those of UX-LLM. UX-LLM demonstrated precision ranging from 0.61 and 0.66 and recall between 0.35 and 0.38, indicating its ability to identify valid usability issues, yet failing to capture the majority of issues. Finally, we conducted a focus group with an app development team of a capstone project developing a transit app for visually impaired persons. The focus group expressed positive perceptions of UX-LLM as it identified unknown usability issues in their app. However, they also raised concerns about its integration into the development workflow, suggesting potential improvements. Our results show that UX-LLM cannot fully replace traditional usability evaluation methods but serves as a valuable supplement particularly for small teams with limited resources, to identify issues in less common user paths, due to its ability to inspect the source code.