# GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent

arXiv: https://arxiv.org/abs/2412.18426 · HTML: https://mlsanity.com/paper/2412.18426/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 47 references that are in the map.
- First posted on arXiv 24 Dec 2024.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Agent Workflow Memory](https://mlsanity.com/paper/2409.07429/) (2024)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents](https://mlsanity.com/paper/2408.07199/) (2024)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Android in the Zoo: Chain-of-Action-Thought for GUI Agents](https://mlsanity.com/paper/2403.02713/) (2024)
- [CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation](https://mlsanity.com/paper/2402.11941/) (2024)
- [Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus](https://mlsanity.com/paper/2209.14927/) (2022)
- [Learning to Navigate the Web](https://mlsanity.com/paper/1812.09195/) (2018)
- [Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing](https://mlsanity.com/paper/2305.09434/) (2023)
- [AUITestAgent: Automatic Requirements Oriented GUI Function Testing](https://mlsanity.com/paper/2407.09018/) (2024)
- [Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps](https://mlsanity.com/paper/2407.03037/) (2024)

## Built on by (most cited first, 11 of 11)

- [A Survey on (M)LLM-Based GUI Agents](https://mlsanity.com/paper/2504.13865/) (2025)
- [Towards Trustworthy GUI Agents: A Survey](https://mlsanity.com/paper/2503.23434/) (2025)
- [SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints](https://mlsanity.com/paper/2509.09853/) (2025)
- [FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents](https://mlsanity.com/paper/2506.01520/) (2025)
- [MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions](https://mlsanity.com/paper/2508.09057/) (2025)
- [GUITester: Enabling GUI Agents for Exploratory Defect Discovery](https://mlsanity.com/paper/2601.04500/) (2026)
- [WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements](https://mlsanity.com/paper/2602.11724/) (2026)
- [WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing](https://mlsanity.com/paper/2603.25226/) (2026)
- [Agent+P: Guiding UI Agents via Symbolic Planning](https://mlsanity.com/paper/2510.06042/) (2025)
- [GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing](https://mlsanity.com/paper/2605.29532/) (2026)
- [AnchorGUI: Asymmetric Memory for Dual-Scale Learning in GUI Navigation](https://mlsanity.com/paper/2609.15457/) (2026)

## Abstract (arXiv)

Nowadays, research on GUI agents is a hot topic in the AI community. However, current research focuses on GUI task automation, limiting the scope of applications in various GUI scenarios. In this paper, we propose a formalized and comprehensive environment to evaluate the entire process of automated GUI Testing (GTArena), offering a fair, standardized environment for consistent operation of diverse multimodal large language models. We divide the testing process into three key subtasks: test intention generation, test task execution, and GUI defect detection, and construct a benchmark dataset based on these to conduct a comprehensive evaluation. It evaluates the performance of different models using three data types: real mobile applications, mobile applications with artificially injected defects, and synthetic data, thoroughly assessing their capabilities in this relevant task. Additionally, we propose a method that helps researchers explore the correlation between the performance of multimodal language large models in specific scenarios and their general capabilities in standard benchmark tests. Experimental results indicate that even the most advanced models struggle to perform well across all sub-tasks of automated GUI Testing, highlighting a significant gap between the current capabilities of Autonomous GUI Testing and its practical, real-world applicability. This gap provides guidance for the future direction of GUI Agent development. Our code is available at https://github.com/ZJU-ACES-ISE/ChatUITest.