# GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies

arXiv: https://arxiv.org/abs/2506.14477 · HTML: https://mlsanity.com/paper/2506.14477/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 19 of 19)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [WebWalker: Benchmarking LLMs in Web Traversal](https://mlsanity.com/paper/2501.07572/) (2025)
- [GUI Agents: A Survey](https://mlsanity.com/paper/2412.13501/) (2024)
- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [WebVLN: Vision-and-Language Navigation on Websites](https://mlsanity.com/paper/2312.15820/) (2023)
- [Human-Centered LLM-Agent User Interface: A Position Paper](https://mlsanity.com/paper/2405.13050/) (2024)

## Built on by (most cited first, 15 of 15)

- [Scaling Agents for Computer Use](https://mlsanity.com/paper/2510.02250/) (2025)
- [MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution](https://mlsanity.com/paper/2601.19199/) (2026)
- [InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2508.19679/) (2025)
- [D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies](https://mlsanity.com/paper/2511.16590/) (2025)
- [MobileBench-OL: A Comprehensive Chinese Benchmark for Evaluating Mobile GUI Agents in Real-World Environment](https://mlsanity.com/paper/2601.20335/) (2026)
- [How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors](https://mlsanity.com/paper/2601.21961/) (2026)
- [AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions](https://mlsanity.com/paper/2605.25707/) (2026)
- [GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models](https://mlsanity.com/paper/2604.14262/) (2026)
- [Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents](https://mlsanity.com/paper/2605.29447/) (2026)
- [Benchmarking and Improving GUI Agents in High-Dynamic Environments](https://mlsanity.com/paper/2604.25380/) (2026)
- [CUJBench: Benchmarking LLM-Agent on Cross-Modal Failure Diagnosis from Browser to Backend](https://mlsanity.com/paper/2604.23455/) (2026)
- [AndroidReality: How Far Are Mobile Agents from the Real World?](https://mlsanity.com/paper/2608.07775/) (2026)
- [Software Engineering for and with GUI Agent](https://mlsanity.com/paper/2608.09278/) (2026)
- [OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents](https://mlsanity.com/paper/2604.24348/) (2026)
- [Cross-View Correspondence Is a Measurement Intervention: Two-Sided Validation for Agent Evaluation and Credit Assignment](https://mlsanity.com/paper/2608.17713/) (2026)

## Abstract (arXiv)

The development of high-quality datasets is crucial for benchmarking and advancing research in Graphical User Interface (GUI) agents. Despite their importance, existing datasets are often constructed under idealized conditions, overlooking the diverse anomalies frequently encountered in real-world deployments. To address this limitation, we introduce GUI-Robust, a novel dataset designed for comprehensive GUI agent evaluation, explicitly incorporating seven common types of anomalies observed in everyday GUI interactions. Furthermore, we propose a semi-automated dataset construction paradigm that collects user action sequences from natural interactions via RPA tools and then generate corresponding step and task descriptions for these actions with the assistance of MLLMs. This paradigm significantly reduces annotation time cost by a factor of over 19 times. Finally, we assess state-of-the-art GUI agents using the GUI-Robust dataset, revealing their substantial performance degradation in abnormal scenarios. We anticipate that our work will highlight the importance of robustness in GUI agents and inspires more future research in this direction. The dataset and code are available at https://github.com/chessbean1/GUI-Robust..