# GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing

arXiv: https://arxiv.org/abs/2605.29532 · HTML: https://mlsanity.com/paper/2605.29532/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 22 references that are in the map.
- First posted on arXiv 28 May 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 14 of 14)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty](https://mlsanity.com/paper/2507.16806/) (2025)
- [AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories](https://mlsanity.com/paper/2504.08942/) (2025)
- [Owl Eyes: Spotting UI Display Issues via Visual Understanding](https://mlsanity.com/paper/2009.01417/) (2020)
- [GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent](https://mlsanity.com/paper/2412.18426/) (2024)
- [Automated Test Input Generation for Android: Are We There Yet?](https://mlsanity.com/paper/1503.07217/) (2015)
- [Nighthawk: Fully Automated Localizing UI Display Issues via Visual Understanding](https://mlsanity.com/paper/2205.13945/) (2022)
- [GUITester: Enabling GUI Agents for Exploratory Defect Discovery](https://mlsanity.com/paper/2601.04500/) (2026)
- [Scenario-Guided LLM-based Mobile App GUI Testing](https://mlsanity.com/paper/2506.05079/) (2025)
- [Navigating the growing field of research on AI for software testing -- the taxonomy for AI-augmented software testing and an ontology-driven literature survey](https://mlsanity.com/paper/2506.14640/) (2025)
- [An Empirical Analysis of UI-based Flaky Tests](https://mlsanity.com/paper/2103.02669/) (2021)

## Abstract (arXiv)

Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation falls short on two fronts. First, existing benchmarks focus almost exclusively on interaction defects, leaving display defects outside the evaluation frame. Second, evaluation protocols are bound to predefined defect annotations, collapsing the testing process into a single end-state judgment that conflates qualitatively distinct failure modes. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types, and introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation beyond predefined annotations, substantially outperforming all baselines. Benchmarking on GUITestScape further reveals that detection remains the critical bottleneck for existing models across both defect types, and that integrating GUIJudge's verifiers into existing agents significantly boosts their detection performance without retraining.